:W&Bá¾_]XÆM 0h3k07stª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0ØØyû‚" t63klpl42mi0 0h3k07stŠ× 0h3k07stjuju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.02741555934662916 epochs‚10  hidden_dim‚128 ( learning_rate‚0.00016868660678080857  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚6  num_encoder_layers‚6Bjjtttt_vanilla_0h3k07stb88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8Š ó£äÔ ¤ÇÚ3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64×[‰E7 0h3k07st&  _runtime‚0  _wandb.runtime‚0W jÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 ò£äÔðžò·*-of*=/data/spatop/options_files/sweep_tttt_jj/config_0h3k07st.json*-l* /scratch/logs*-n*jjtttt_vanilla_0h3k07st2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8r/opt/conda/bin/pythonº ykpn1bd1mizct134cxvu1o2jqk3e1d5wWÜ|ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€ «”婢àãÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-78f24734-e3a1-59a7-ce1b-bb4283f77c97Ê13.2º ykpn1bd1mizct134cxvu1o2jqk3e1d5wµª½Ç2  requirements.txt2'·W 0h3k07stZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64!ýRøY 0h3k07stZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64»¼ú*  0h3k07stb trainer/global_step2ýnÌž\  0h3k07stZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64‡"/  0h3k07stb*"trainer/global_step2»omPK  0h3k07stj: ô£äÔ ³«¤*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ™Â²C“  0h3k07stj  ô£äÔ Öö’î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 2.0 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 38.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ŸˆÕŽ\ 0h3k07stjÊ ô£äÔ¸àí“·Trainable params: 40.9 M Non-trainable params: 86 Total params: 40.9 M Total estimated model params size (MB): 163.524 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 Ypkq 0h3k07stj` ô£äÔ€„„”Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). «ã>{ 0h3k07stjj ô£äÔ·”Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). S7£'y 0h3k07stjh ô£äÔà±””Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). wc€ 0h3k07stjo ô£äÔ˜¢¡”_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Z)«.z 0h3k07stji ô£äÔ€†­”Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). Æç}t 0h3k07stjc ô£äÔ🴔Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). E(K¶q 0h3k07stj` ô£äÔˆƒ»”Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ¶Œ’ t 0h3k07stjc ô£äÔø“Å”Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). w¦›5n 0h3k07stj] ô£äÔðÎΔMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Wz¶pm 0h3k07stj\ ô£äÔÈçØ”Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). Œ1÷ßb 0h3k07st*Ð  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"âjë 0h3k07stjÙ ô£äÔú»™È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ¤Ë„N 0h3k07stj¼ ô£äÔػș«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ìýj°ê 0h3k07stjØ ÷£äÔÀº¯RÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ]ÇŒ+O 0h3k07stj½ ÷£äÔø¡¿R­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. €Ý…‘‚(:‡ ƒ¤äÔ¼µ ~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î7=¢,‚(:¢ ƒ¤äÔ‡¼„ network.recv‚38995 disk.drbd2080.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 44.62890625 disk.sda3.out‚ 0.87109375 disk.dm-1.out‚ 0.87109375 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0) proc.memory.availableMB‚ 2677.25390625* proc.memory.percent‚16.329312324523926 disk.drbd1716.in‚0 network.sent‚33597 disk.drbd2266.in‚0 disk.sda.out‚ 0.87109375 disk.drbd2265.in‚0* disk./.usagePercent‚16.898301600960526 disk.dm-0.out‚ 44.62890625 disk.sda1.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2675.39453125 disk.rbd1.in‚0$ memory_percent‚83.65933895111084 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 44.62890625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚0% disk./.usageGB‚251.79690551757812 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.sda.in‚0 disk.drbd2265.out‚0 cpu‚25.238328983674457u2*‘ ‚(:‡ ’¤äÔý¯~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.323' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2646 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨P"%!‚(:› ’¤äÔø§€) disk./.usagePercent‚16.89890270665092 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚0' proc.memory.availableMB‚ 2675.171875 disk.nvme0n1.out‚ 97.80078125" disk.nvme0c0n1.out‚ 97.80078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2675.41796875 cpu‚25.187474210570315 disk.nvme0c0n1.in‚0.078125 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1.4296875 disk.drbd2266.in‚0* proc.memory.percent‚16.329455375671387 proc.cpu.threads‚17 disk.sda.out‚ 1.4296875 disk.nvme0n1.in‚0.078125 disk.sda3.in‚0 disk.sda3.out‚ 1.4296875 disk.drbd2265.out‚0$ memory_percent‚83.67204666137695$ disk./.usageGB‚251.8058624267578 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚50780 network.recv‚57459 disk.sda.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 97.80078125—;Äa3"b/ lr-AdamW/pg1"trainer/global_step2:JP<Ñsˆ3#b/ lr-AdamW/pg2"trainer/global_step2:JPÅß -$) _wandbruntime‚39  _runtime‚39†’¾%¹ _runtime‚ 39.93223011 _step‚0 & lr-AdamW/pg1‚6.836760738014574e-06 & lr-AdamW/pg2‚6.836760738014574e-06 trainer/global_step‚49 $ _timestamp‚1.788416540194531e+09"ÆkÝÀ&» & lr-AdamW/pg1‚6.836760738014574e-06 & lr-AdamW/pg2‚6.836760738014574e-06 trainer/global_step‚49 $ _timestamp‚1.788416540194531e+09 _runtime‚ 39.93223011 _step‚0vÈBl'bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPiÞc±f(bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPŽ47%j)bf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP]T:éi*be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPìä†o+bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP™`¼’p,bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPòY4s-bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP´8i˜n.bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPv=NXr/bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPx«hs0bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP–,ª8h1bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPܳ%Èe2ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP¸Ð5Pm3bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP‘¾`i4be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP‚h^5bZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPð2·d6b` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JP|‹@…x7bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP;Oëg8bc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP'!UÙf9bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP®p«f:bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP4×T];bY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPhëæˆmbg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPé·áƒg?bc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP kËjh@bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP³@Ý­mAbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP®màávBbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPAE°…hCbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPÑ\äKqDbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPK!jEbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPé-§þtFbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPNBê*kGbg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP_¡¼]pHbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP_~?qiIbe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP=¤“{pJbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPNôHnKbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÂʯmoLbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPÓ¿E­nMbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPåg9qNbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPŸœ†ÒlObh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPr—¨iPbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP¯³=OqQbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPQ^½gRbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPÛÕ¯$aSb] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP¾U€ÜnTbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP—íÿ6iUbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP«nvbVb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPr»#ImWbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPYZ–¢oXbk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP§F^YbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JPÛ¶J¾iZbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP´Ñ½__[b[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JPŽOÿÄe\ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP]F¡Ak]bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP;žz—n^bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP“L'Ol_bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPB9¶Zh`bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP–Fнfabb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP.¨çÛlbbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JP0B +ncbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP!î„kfdbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP½ïˆ´nebj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP5~’gfbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JPò‰Úhgbd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP4ÔÏ—shbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP)º¬qribn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP)-|bjb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPh—¯`kb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPýHN_lb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPI§»kmbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÓIµsgnbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP(‚~gsobo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP<¸¸\jpbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP˜apqbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPw%¤mrbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPfÒªcsb_ !¡jbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JP H•k€bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPÈ>IQobj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPJŒt"f‚ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPiéûörƒbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPX!Õo„bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPbÆBl…bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÝ¢Rw†br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP¼³Ua‡b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JP"G hˆbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPµì}y‰bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP-´VThŠbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPv81f‹ba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP|+ÓciŒbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP0,uÂfba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP5¬«lŽbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPx^Œðbb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JP¯ôlKubp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPâ$'¤r‘bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPŒÖ…2o’bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPX ]e“b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPKÛèSm”bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPUq9¤k•bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP °ö&d–b_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884165401953182e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 trainer/global_step‚49 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40735264768 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚645130 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 _step‚1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2875955288576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863047680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚835 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 43352637440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2875281690112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚939738 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚645582 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3103192194560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 43352637440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚843 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2875955288576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884165401953182e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954818048 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 43517028352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691469 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚413 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2875281690112 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 trainer/global_step‚49 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚708398 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3103168254464 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 866123776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 2834564428288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 40735264768 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚414100 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3146709222912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚645130 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845965 _step‚1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2875955288576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22901 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1629594 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚921198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚708396 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863047680 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚1648136 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2801473167074 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 41237680866 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚843 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚843 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863047680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954818048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 2834564428288 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚835 HDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP‹¿•\©bW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPl ÕÏaªb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP1Ð9“b«b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPG5ÖBp¬bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPX7)g­bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPË´›„j®be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP¬Գn¯bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP” Ägg°bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPQÑ‹vc±b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPlª—³o²bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP“% ó_³bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPûo¹bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPÛ¼.pºbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP Kl»bg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP8en¼bi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPrr;2l½bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP]ÿ­ae¾b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JPàÏÖk¿bf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP£T›mÀbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP‹¶7$^ÁbY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPƒ##X`Âb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPn`N9mÃbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPAœ°biÄbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPu6¹4sÅbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP"nÆbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÃ>N¡eÇb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPbdÃYfÈba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP÷ÚL=iÉbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPWÐÖ!nÊbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPû_NerËbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPMÕ’ÅdÌb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPÄÕ¬œrÐbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP½ìˆÇnÑbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP RAÞgÒbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPöÑß/fÓba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP-•kjÔbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JP#àFhÕbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JP(ñ…jÖbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP ¨òq×bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP(æ¬cgØbb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JPRTÅpÙbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP¤F”]rÚbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÛÓØNhÛbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPFêBaÜb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP äKrÝbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPI¼çkÞbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÂÜÅmßbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPæf®»bàb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JP5,™—rábm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP‚Ktu^âbY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP§ã•cãb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP9¢C]käbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPúU†,iåbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPÜc+oæbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPo]›«]çbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP”a^Údèb_ yijëbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPì]¹‡lìbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPbj4jíbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPõçŽeîb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPØ“Òmïbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP·s ×mðbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP‚… DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÂzí½uúbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPüxŠîsûbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPþ…·ýhübc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPäj`«iýbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPÜM&gbþb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPh /Àfÿba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP4·Öe€b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPO 6ibd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP o‚bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP0¸™Æwƒbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPTçYS^„bY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP Îp´h…bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPÉL¡m†bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPo,ck‡bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPꊿlˆbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP¨B÷·p‰bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPšº‹°eŠb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP%¸uj‹be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPr*>HqŒbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPþÐÁ„eb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP)Âù³`Žb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÔÃ\db_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP™L¼¢m”bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP;~Xm•bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPŸÌ’åh–bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP &]—bX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPüvXWd˜b_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP|‚¹ržbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP FmŸbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP³É ó. ) _wandbruntime‚40  _runtime‚40Ej#¹D¡‡M LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884165408810825e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 44205270528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 _runtime‚ 40.619244911 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.al® …TIlocated‚843 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2857193732598 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚236085 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192ƒ(½­‘&¢‹M >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚722414 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3209216774656 _runtime‚ 40.619244911 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚863 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚413 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚235630 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1680632 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 42054116038 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2933127768576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚657898 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚843 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1680632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2932454156800 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚658355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚722412 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3164818992128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2815794046976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10192158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2890899464192 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 42210301440 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2890917467136 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 41554692608 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2857193732598 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚236085 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 181444096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚422268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 2815811086336 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1662064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 2890899464192 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚939652 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884165408810825e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 44205270528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚722414 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1662064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚958218 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 42210301440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3164842932224¢”ÈÅC£b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP]S~«@¤b; loss/FBt2/detection_loss"trainer/global_step2:JP#iØA¥b< loss/FRt2/assignment_loss"trainer/global_step2:JP,A¦b< loss/FRt3/assignment_loss"trainer/global_step2:JPiRyÀC§b> loss/SRqqt4/assignment_loss"trainer/global_step2:JPª‡B¨b= loss/SRqqt1/detection_loss"trainer/global_step2:JPnaNØB©b= loss/SRqqt4/detection_loss"trainer/global_step2:JP6ðÌ6Aªb< loss/FRt1/assignment_loss"trainer/global_step2:JPd.¡ÄA«b< loss/FBt2/assignment_loss"trainer/global_step2:JPüûù[A¬b< loss/FBt3/assignment_loss"trainer/global_step2:JPÎá¬B­b= loss/SRqqt3/detection_loss"trainer/global_step2:JPl‘Éé7®b2 loss/total_loss"trainer/global_step2:JPÂØA¯b< loss/FRt4/assignment_loss"trainer/global_step2:JP§‹l C°b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPL+ä|@±b; loss/FRt2/detection_loss"trainer/global_step2:JP§â\Ò@²b; loss/FBt1/detection_loss"trainer/global_step2:JPÐ.G@³b; loss/FBt3/detection_loss"trainer/global_step2:JPlC ´A´b< loss/FBt1/assignment_loss"trainer/global_step2:JPn©j@µb; loss/FRt3/detection_loss"trainer/global_step2:JP-¾åÆB¶b= loss/SRqqt2/detection_loss"trainer/global_step2:JPߘ-·b( epoch"trainer/global_step2:JPmðƒ«C¸b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP|˜*@¹b; loss/FBt4/detection_loss"trainer/global_step2:JPH¯¿{Aºb< loss/FBt4/assignment_loss"trainer/global_step2:JP£Õ@»b; loss/FRt1/detection_loss"trainer/global_step2:JPj^ÑÈ@¼b; loss/FRt4/detection_loss"trainer/global_step2:JPå¸{–.½) _wandbruntime‚40  _runtime‚40¨5ÔÆL¾Æ /loss/FRt3/detection_loss‚4.1945429429411885 1loss/SRqqt2/detection_loss‚3.4423953396081926 epoch‚0 % _timestamp‚1.7884165408829558e+09 1loss/SRqqt3/assignment_loss‚5.336389390249101 /loss/FBt4/detection_loss‚6.3053281991570085 _step‚3 0loss/FBt4/assignment_loss‚2.7228655020395913 .loss/FRt1/detection_loss‚3.750632288672666 /loss/FRt4/detection_loss‚3.9364001218761717 1loss/SRqqt1/assignment_loss‚5.531241347573021 /loss/FBt2/detection_loss‚4.2810569823616085 /loss/FRt2/assignment_loss‚7.303034579753875 /loss/FRt3/assignment_loss‚7.395180666446686 1loss/SRqqt4/assignment_loss‚5.479332810356503 1loss/SRqqt1/detection_loss‚2.9892927169799806 0loss/SRqqt4/detection_loss‚4.225713333913258 _runtime‚ 40.620395864 0loss/FRt1/assignment_loss‚7.3771466045844845 /loss/FBt2/assignment_loss‚2.760816646946801 /loss/FBt3/assignment_loss‚2.773308808986957 1loss/SRqqt3/detection_loss‚2.3178040735305303 &loss/total_loss‚118.05544663499047 /loss/FRt4/assignment_loss‚7.307161626361665 1loss/SRqqt2/assignment_loss‚5.437363176345825 .loss/FRt2/detection_loss‚4.250995946675539 .loss/FBt1/detection_loss‚9.517007588512367 .loss/FBt3/detection_loss‚6.643221387496362 0loss/FBt1/assignment_loss‚2.7772145536210804 trainer/global_step‚49ô`DP¿Ê 0loss/FRt1/assignment_loss‚7.3771466045844845 /loss/FBt2/assignment_loss‚2.760816646946801 /loss/FBt3/assignment_loss‚2.773308808986957 1loss/SRqqt3/detection_loss‚2.3178040735305303 &loss/total_loss‚118.05544663499047 /loss/FRt4/assignment_loss‚7.307161626361665 1loss/SRqqt2/assignment_loss‚5.437363176345825 .loss/FRt2/detection_loss‚4.250995946675539 .loss/FBt1/detection_loss‚9.517007588512367 .loss/FBt3/detection_loss‚6.643221387496362 0loss/FBt1/assignment_loss‚2.7772145536210804 trainer/global_step‚49 /loss/FRt3/detection_loss‚4.1945429429411885 1loss/SRqqt2/detection_loss‚3.4423953396081926 epoch‚0 % _timestamp‚1.7884165408829558e+09 1loss/SRqqt3/assignment_loss‚5.336389390249101 /loss/FBt4/detection_loss‚6.3053281991570085 _step‚3 0loss/FBt4/assignment_loss‚2.7228655020395913 .loss/FRt1/detection_loss‚3.750632288672666 /loss/FRt4/detection_loss‚3.9364001218761717 1loss/SRqqt1/assignment_loss‚5.531241347573021 /loss/FBt2/detection_loss‚4.2810569823616085 /loss/FRt2/assignment_loss‚7.303034579753875 /loss/FRt3/assignment_loss‚7.395180666446686 1loss/SRqqt4/assignment_loss‚5.479332810356503 1loss/SRqqt1/detection_loss‚2.9892927169799806 0loss/SRqqt4/detection_loss‚4.225713333913258 _runtime‚ 40.620395864Äv®À‚(:’ ¡¤äÔŽËõ} gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚35.0 gpu.0.powerWatts‚132.148' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.429599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÕ~3Á‚(:¨ ¡¤äÔ݉‡€ disk.rbd1.in‚0 disk.nvme0n1.in‚0.078125* proc.memory.percent‚16.329622268676758% disk.nvme0c0n1.out‚143.4228515625$ memory_percent‚83.68561267852783 network.sent‚63962) disk./.usagePercent‚16.89985044654268 disk.nvme0c0n1.in‚0.078125 disk.sda2.in‚0 disk.sda3.out‚ 1.95703125 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 1.95703125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda1.in‚0# disk.nvme0n1.out‚143.4228515625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2675.4453125 proc.cpu.threads‚17 network.recv‚75323 disk.dm-1.out‚ 1.95703125 disk.drbd2266.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.sda.in‚0% disk./.usageGB‚251.81998443603516 disk.drbd2265.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2672.94921875 cpu‚25.15867531892387 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚143.4228515625{¬GœÂ‚(:’ °¤äÔˆãŒ~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)ó^E4Â(:© °¤äÔá»â€ network.sent‚127522 disk.dm-0.in‚0.078125 disk.dm-1.out‚ 2.5859375 disk.nvme0c0n1.in‚0.078125) disk./.usagePercent‚16.90070192760156& disk.nvme0c0n1.out‚211.54736328125 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.078125 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ disk.nvme0n1.out‚211.54736328125 disk.rbd0.in‚0* proc.memory.percent‚16.329669952392578 cpu‚25.13641141602626 network.recv‚107132 disk.sda.in‚0 disk.sda.out‚ 2.5859375% disk./.usageGB‚251.83267211914062 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 2671.91796875! proc.memory.rssMB‚ 2675.453125! disk.dm-0.out‚211.54736328125 disk.rbd1.in‚0 disk.rbd1.out‚0 disk.sda3.out‚ 2.5859375$ memory_percent‚83.691906929016114÷â’Ä‚(:‡ ¿¤äÔ¸§~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.643' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,&Ê:Å‚(:¯ ¿¤äÔéÀ€) proc.memory.percent‚16.32969379425049 network.recv‚112224* disk./.usagePercent‚16.899471299384466 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚134367 disk.drbd2080.in‚0 disk.rbd1.out‚0$ memory_percent‚84.44221019744873 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.rbd1.in‚0 disk.sda.out‚ 3.13671875 disk.nvme0c0n1.in‚0.078125 cpu‚25.123735320921988 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚270.43798828125 proc.cpu.threads‚17 disk.rbd0.in‚0% disk./.usageGB‚251.81433486938477 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 3.13671875! disk.dm-0.out‚270.43798828125 disk.dm-1.out‚ 3.13671875 disk.sda.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚270.43798828125 disk.sda2.in‚0) proc.memory.availableMB‚ 2548.98828125# proc.memory.rssMB‚ 2675.45703125ç3Uw.Æ) _wandbruntime‚76  _runtime‚76ú+„ ÂǼ ' lr-AdamW/pg1‚1.3813047205376384e-05 ' lr-AdamW/pg2‚1.3813047205376384e-05 trainer/global_step‚99 % _timestamp‚1.7884165770685184e+09 _runtime‚ 76.80610201 _step‚4:a¦.ÆÈÀ trainer/global_step‚99 % _timestamp‚1.7884165770685184e+09 _runtime‚ 76.80610201 _step‚4 ' lr-AdamW/pg1‚1.3813047205376384e-05 ' lr-AdamW/pg2‚1.3813047205376384e-05&·³·.É) _wandbruntime‚76  _runtime‚76ô=Ö[e#ʇO \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 82073020274 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10192158720 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5587515034482 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1282161 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚822500 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚5 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚822502 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884165770692785e+09 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5506096444416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884165770692785e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5733919438336 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5652216370688 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 81721070592 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5734593036800 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3274988 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6185705137664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10192158720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 332015104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151808000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 82376666112 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚5 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 866123776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6271899681792 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 866123776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 6185729077760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚459222 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚413 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚863 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚822502 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5588186504050 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1409196 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10192158720 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1409196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0î²»o.Ì) _wandbruntime‚77  _runtime‚77ÇÉŸ—&Í‘M ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3288916 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5643907069782 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚413 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 83196107264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚830668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5708551406592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884165777656033e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5561000574976 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1865704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5791091905024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 87216459776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 866123776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192512000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6334596275200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚6 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 77.503710709 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 82540498432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚99 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3307484 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚843 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10192158720 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1884270 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1423214 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6334596275200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚6 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 82540498432 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 82235025030 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 77.503710709 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 82540498432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 trainer/global_step‚99 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1294420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚843 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5561017614336 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 83196107264 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3307484 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚843 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1884270 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 866123776 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5791765516800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚830668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6247379815424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1423212 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10192158720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5708551406592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚413 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1865704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884165777656033e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5791765516800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5708569409536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5708551406592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6247355875328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5561000574976 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3288916 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5708569409536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1865704 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚464205 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0{ÌpP.Ï) _wandbruntime‚77  _runtime‚77wO»þCн /loss/FRt1/assignment_loss‚7.370223637284903 0loss/FBt1/assignment_loss‚2.6221214532852173 .loss/FRt1/detection_loss‚5.469631767793789 0loss/SRqqt2/detection_loss‚2.915497179143131 -loss/FBt3/detection_loss‚4.37570847169711 .loss/FBt4/detection_loss‚9.142403324967937 &loss/total_loss‚123.34837353166986 trainer/global_step‚99 /loss/FRt3/assignment_loss‚7.261527217667679 _step‚7 1loss/SRqqt3/assignment_loss‚5.360561964246962 0loss/FBt3/assignment_loss‚2.6344223022460938 /loss/FBt1/detection_loss‚10.799097146838903 .loss/FBt2/detection_loss‚4.052000152464542 /loss/FRt2/assignment_loss‚7.232370731442473 1loss/SRqqt4/assignment_loss‚5.399636796542576 /loss/FBt2/assignment_loss‚2.610963741938273 .loss/FRt3/detection_loss‚2.663845136633207 0loss/SRqqt4/detection_loss‚6.371293433010578 epoch‚0 1loss/SRqqt1/detection_loss‚2.7392005341356263 1loss/SRqqt2/assignment_loss‚5.406466330800738 /loss/FRt2/detection_loss‚3.7511716706808222 -loss/FRt4/detection_loss‚6.21365211797612 0loss/SRqqt3/detection_loss‚3.416287386086252 _runtime‚ 77.50516788 .loss/FRt4/assignment_loss‚7.34591897896358 1loss/SRqqt1/assignment_loss‚5.532375026167485 /loss/FBt4/assignment_loss‚2.661997029655858 % _timestamp‚1.7884165777677164e+096àµl>ÑÁ % _timestamp‚1.7884165777677164e+09 /loss/FRt1/assignment_loss‚7.370223637284903 0loss/FBt1/assignment_loss‚2.6221214532852173 .loss/FRt1/detection_loss‚5.469631767793789 0loss/SRqqt2/detection_loss‚2.915497179143131 -loss/FBt3/detection_loss‚4.37570847169711 .loss/FBt4/detection_loss‚ŸÈ6 9.142403324967937 &loss/total_loss‚123.34837353166986 trainer/global_step‚99 /loss/FRt3/assignment_loss‚7.261527217667679 _step‚7 1loss/SRqqt3/assignment_loss‚5.360561964246962 0loss/FBt3/assignment_loss‚2.6344223022460938 /loss/FBt1/detection_loss‚10.799097146838903 .loss/FBt2/detection_loss‚4.052000152464542 /loss/FRt2/assignment_loss‚7.232370731442473 1loss/SRqqt4/assignment_loss‚5.399636796542576 /loss/FBt2/assignment_loss‚2.610963741938273 .loss/FRt3/detection_loss‚2.663845136633207 0loss/SRqqt4/detection_loss‚6.371293433010578 epoch‚0 1loss/SRqqt1/detection_loss‚2.7392005341356263 1loss/SRqqt2/assignment_loss‚5.406466330800738 /loss/FRt2/detection_loss‚3.7511716706808222 -loss/FRt4/detection_loss‚6.21365211797612 0loss/SRqqt3/detection_loss‚3.416287386086252 _runtime‚ 77.50516788 .loss/FRt4/assignment_loss‚7.34591897896358 1loss/SRqqt1/assignment_loss‚5.532375026167485 /loss/FBt4/assignment_loss‚2.661997029655858~À¦1ŽÒ‚(:ƒ ΤäÔ糚~ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.88 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/2϶6Ó‚(:« ΤäÔ˺å disk.drbd1716.in‚0 disk.sda3.out‚ 3.94140625 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 network.sent‚141431 disk.sda2.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0( proc.memory.percent‚16.3297176361084* disk./.usagePercent‚16.900319964360474 disk.drbd1716.out‚0$ memory_percent‚84.44387912750244 cpu‚25.114533546636423 proc.cpu.threads‚17$ disk./.usageGB‚251.8269805908203 disk.drbd2265.in‚0 disk.rbd1.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 2548.71484375 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.dm-1.in‚0 network.recv‚117743& disk.nvme0c0n1.out‚326.41650390625 disk.drbd2080.in‚0! disk.dm-0.out‚326.41650390625 disk.dm-1.out‚ 3.94140625 disk.drbd2266.in‚0 disk.sda.out‚ 3.94140625$ disk.nvme0n1.out‚326.41650390625 disk.drbd2266.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2675.4609375˜Ô³’Ô‚(:‡ ݤäÔÿÂ~ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.502517700195312* gpu.0.memoryAllocatedBytes‚ 10739580928 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìï7Õ‚(:¬ ݤäÔÕš€ disk.sda.in‚0 disk.sda.out‚ 4.34765625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚0 disk.nvme0c0n1.in‚0.078125) proc.memory.percent‚16.32974147796631 network.recv‚123938$ disk.nvme0n1.out‚373.60400390625& disk.nvme0c0n1.out‚373.60400390625 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0! disk.dm-0.out‚373.60400390625 cpu‚25.10508050286048* disk./.usagePercent‚16.898772398815993 disk.sda3.out‚ 4.34765625 disk.dm-0.in‚0.078125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 4.34765625$ memory_percent‚84.44561958312988 network.sent‚191266$ disk./.usageGB‚251.8039207458496 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚0.078125 disk.rbd0.out‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 2548.4296875# proc.memory.rssMB‚ 2675.46484375 proc.cpu.threads‚17Â~ó0Ö+ _wandbruntime‚113  _runtime‚113I}rúÄ×¾ _step‚8 ' lr-AdamW/pg1‚2.0789333672738193e-05 ' lr-AdamW/pg2‚2.0789333672738193e-05 trainer/global_step‚149 % _timestamp‚1.7884166140882223e+09 _runtime‚ 113.82576061µB½þÈØ ' lr-AdamW/pg1‚2.0789333672738193e-05 ' lr-AdamW/pg2‚2.0789333672738193e-05 trainer/global_step‚149 % _timestamp‚1.7884166140882223e+09 _runtime‚ 113.82576061 _step‚8É£JË0Ù+ _wandbruntime‚113  _runtime‚113RêL&™'Ú“O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 123362471936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 122706876416 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2109998 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 122253929474 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8592557186560 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2773302 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8469868313088 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4883298 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8374228371458 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9397364873728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2109998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2791842 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1918787 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2109996 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1230900 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8251320011776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚9 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 _runtime‚ 113.826693501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8469868313088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 190428160 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2791842 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1918787 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 9268265960960 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 122908359682 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚687885 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9397555301888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 8469850310144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2109996 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1230900 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 8251320011776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚9 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 129289340928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 122706876416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8373556901890 _runtime‚ 113.826693501 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8592557186560 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8469868313088 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 129122852864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚149 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2773302 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2109996 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2791842 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1918347 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8593230785024 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 9268242020864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8593230785024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ‘ü0Ü+ _wandbruntime‚114  _runtime‚114ú¼á½ÝžM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚692342 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2124014 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 130146810368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8306224142336 % _timestamp‚1.7884166147774158e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚10 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8526203348992 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8526221351936 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8649729653248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8650403265024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 F:DeviceStatsMonitor.on_train_batch_end/allocation.aÌÛ:çll.freed‚4915768 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4934336 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚149 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256y¨}‚¨&Þ¢M NDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 130146810368 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 8306224142336 % _timestamp‚1.7884166147774158e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8526203348992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2124012 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2810322 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1239068 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚10 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 129974044160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8526203348992 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 124181913088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1239070 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2124012 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8526221351936 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8649729653248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8650403265024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 8526221351936 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1930972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8650403265024 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4915768 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9459866802688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9329916698624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8429277467414 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4934336 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚149 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2810322 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 123526304256 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚691904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2791756 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 123526304256 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 124181913088 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4934336 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚692342 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2124014 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 8306207102976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >€w³0ß+ _wandbruntime‚114  _runtime‚114:ø/kHà epoch‚0 trainer/global_step‚149 % _timestamp‚1.7884166147790117e+09 0loss/FRt2/assignment_loss‚7.3788785593850275 /loss/FRt4/assignment_loss‚7.366698443889618 1loss/SRqqt2/assignment_loss‚5.386567057789983 .loss/FBt2/detection_loss‚1.793231973645934 %loss/total_loss‚128.7251444983495 _step‚11 /loss/FRt1/detection_loss‚6.6484565071084285 -loss/FRt3/detection_loss‚2.46211933087686 1loss/SRqqt1/detection_loss‚2.5912686210583176 /loss/FBt4/detection_loss‚13.129855362268595 /loss/FBt3/assignment_loss‚2.061374764442444 -loss/FBt1/detection_loss‚19.2235141131613 /loss/FRt1/assignment_loss‚7.266501231627031 /loss/FRt3/assignment_loss‚7.151883857003574 1loss/SRqqt1/assignment_loss‚5.548550538849413 1loss/SRqqt3/assignment_loss‚5.369487255811691 .loss/FBt2/assignment_loss‚2.03157114982605 1loss/SRqqt2/detection_loss‚1.8466130660192386 0loss/SRqqt3/detection_loss‚3.351743509682516 .loss/FBt3/detection_loss‚6.928371949195862 /loss/FBt4/assignment_loss‚2.100514760384193 .loss/FRt4/detection_loss‚4.299746053665876 _runtime‚ 114.516713616 1loss/SRqqt4/assignment_loss‚5.503081810780061 0loss/FBt1/assignment_loss‚2.0403265158335366 /loss/FRt2/detection_loss‚2.6510695900235857 0loss/SRqqt4/detection_loss‚4.593718476020372­çbLáÆ 0loss/FBt1/assignment_loss‚2.0403265158335366 /loss/FRt2/detection_loss‚2.6510695900235857 0loss/SRqqt4/detection_loss‚4.593718476020372 epoch‚0 trainer/global_step‚149 % _timestamp‚1.7884166147790117e+09 0loss/FRt2/assignment_loss‚7.3788785593850275 /loss/FRt4/assignment_loss‚7.366698443889618 1loss/SRqqt2/assignment_loss‚5.386567057789983 .loss/FBt2/detection_loss‚1.793231973645934 %loss/total_loss‚128.7251444983495 _step‚11 /loss/FRt1/detection_loss‚6.6484565071084285 -loss/FRt3/detection_loss‚2.46211933087686 1loss/SRqqt1/detection_loss‚2.5912686210583176 /loss/FBt4/detection_loss‚13.129855362268595 /loss/FBt3/assignment_loss‚2.061374764442444 -loss/FBt1/detection_loss‚19.2235141131613 /loss/FRt1/assignment_loss‚7.266501231627031 /loss/FRt3/assignment_loss‚7.151883857003574 1loss/SRqqt1/assignment_loss‚5.548550538849413 1loss/SRqqt3/assignment_loss‚5.369487255811691 .loss/FBt2/assignment_loss‚2.03157114982605 1loss/SRqqt2/detection_loss‚1.8466130660192386 0loss/SRqqt3/detection_loss‚3.351743509682516 .loss/FBt3/detection_loss‚6.928371949195862 /loss/FBt4/assignment_loss‚2.100514760384193 .loss/FRt4/detection_loss‚4.299746053665876 _runtime‚ 114.516713616 1loss/SRqqt4/assignment_loss‚5.503081810780061 žZf’â‚(:‡ ì¤äÔÞ‹‡~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a+­¼8ã‚(:­ ì¤äÔÈ­ƒ disk.sda1.in‚0 disk.rbd1.in‚0 cpu‚25.100850536685794 network.recv‚129300 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 5.00390625% disk./.usageGB‚251.84175872802734 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.078125 disk.sda3.in‚0 disk.sda.out‚ 5.00390625) proc.memory.percent‚16.32974147796631* disk./.usagePercent‚16.901311737548124 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.sda.in‚0$ memory_percent‚84.44490432739258$ disk.nvme0n1.out‚445.24267578125 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.078125 disk.dm-1.out‚ 5.00390625' proc.memory.availableMB‚ 2548.546875# proc.memory.rssMB‚ 2675.46484375 disk.nvme0n1.in‚0.078125& disk.nvme0c0n1.out‚445.24267578125 disk.drbd2265.in‚0 disk.drbd2265.out‚0! disk.dm-0.out‚445.24267578125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚198066 disk.sda2.in‚0 proc.cpu.threads‚17¤hä‚(:’ û¤äÔ¾™~ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óS%å‚(:š û¤äÔ“É¢ disk.dm-0.out‚ 501.150390625 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚16.901756934650162 disk.drbd2266.out‚0 disk.sda.in‚0 disk.sda.out‚5.640625 disk.sda3.out‚5.640625% disk./.usageGB‚251.84839248657227 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 2548.515625 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.rbd1.out‚0 disk.drbd1716.in‚0$ disk.nvme0c0n1.out‚ 501.150390625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda1.in‚0" disk.nvme0n1.out‚ 501.150390625 disk.rbd0.out‚0 network.sent‚247388 network.recv‚134605 disk.dm-0.in‚0.09375 disk.dm-1.out‚5.640625$ memory_percent‚84.44509506225586 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375 disk.sda3.in‚0) proc.memory.percent‚16.32976531982422 disk.sda2.out‚0 proc.memory.rssMB‚ 2675.46875 cpu‚25.09603517357647§J&I’æ‚(:‡ Š¥äÔâ ‡~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.829' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.5658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µpÔ4ç‚(:© Š¥äÔÈë € disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 network.sent‚253978! disk.dm-0.out‚557.28662109375 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 disk.sda3.out‚ 6.19140625 disk.rbd1.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 2548.4609375 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.out‚0 network.recv‚139646 disk.drbd2265.out‚0& disk.nvme0c0n1.out‚557.28662109375) proc.memory.percent‚16.32978916168213 proc.cpu.threads‚17 disk.dm-0.in‚0.09375 disk.sda.out‚ 6.19140625 disk.drbd1716.in‚0$ disk.nvme0n1.out‚557.28662109375# memory_percent‚84.4454288482666* disk./.usagePercent‚16.900733672535807 disk.dm-1.out‚ 6.19140625 disk.drbd2266.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2675.47265625 cpu‚25.09241189465796% disk./.usageGB‚251.83314514160156ݪƒÑ0è+ _wandbruntime‚150  _runtime‚150ñlý‡Åé¿ trainer/global_step‚199 % _timestamp‚1.7884166511312103e+09 _runtime‚ 150.86871515 _step‚12 ' lr-AdamW/pg1‚2.7765620140100004e-05 ' lr-AdamW/pg2‚2.7765620140100004e-05ZsÉêà _step‚12 ' lr-AdamW/pg1‚2.7765620140100004e-05 ' lr-AdamW/pg2‚2.7765620140100004e-05 trainer/global_step‚199 % _timestamp‚1.7884166511312103e+09 _runtime‚ 150.86871515 r·j0ë+ _wandbruntime‚150  _runtime‚150ºçªíª'ì¤O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11287520255488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 163089268882 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚916031 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 150.869689218 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884166511318948e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11287502252544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12350778904064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11160270238866 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _step‚13 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298²GÃL í¨O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _step‚13 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚855 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 171953641984 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚10996526539776 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚10996509500416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3699354 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11451194934784 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11159598769298 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11287520255488 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 164348277760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11287520255488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚916481 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3717894 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2810798 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1639300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 163743699090 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freŠÆ‹³bed‚ 163089268882 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚916031 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 150.869689218 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 163692682240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884166511318948e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2810798 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 163692682240 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6528692 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚874 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11287502252544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3717894 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2555331 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11451194934784 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2555783 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 172118032896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11287502252544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12350778904064 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11160270238866 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11451868533248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1639302 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2810796 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚199 'Ý0î+ _wandbruntime‚151  _runtime‚151:0ò ³&ï­M HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3736374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 165167718912 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12585235206144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 % _timestamp‚1.7884166518270257e+09 _step‚14 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚199 IH¹·&ð±M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11509041013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 164560134262 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3717808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2824812 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 172978330624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11343855291392 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11343873294336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚855 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12585431912448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 _runtime‚ 151.56510843 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2824814 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2824814 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚920605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12412429641728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196706304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3736374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 164512110080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1647470 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3717808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 165167718912 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2568073 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172766208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1647468 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 164512110080 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6561188 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3736374 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2568533 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11343855291392 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11051413630976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚11508367401472 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12412453581824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12585235206144 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 172805564416 % _timestamp‚1.7884166518270257e+09 _step‚14 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6561188 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14310506281472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14105154194944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13946312106274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 trainer/global_step‚249 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1146821 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3194082 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚ 187.669001296 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3511596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14105172197888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944!p¿|°'ýªO trainer/global_step‚249 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14309832683008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 215005637120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3511596 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4643946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1146821 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3194082 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚14310506281472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚ 187.669001296 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3511596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14105172197888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14105154194944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 204579038498 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14309832683008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚879 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15648345364480 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15648159130624 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 204678488064 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8155544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚860 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 204678488064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14310506281472 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14105154194944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15433339727360 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15433315787264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047700 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3511598 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13946312106274 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973€ÓDX0þ+ _wandbruntime‚188  _runtime‚188¾jgµ&ÿ¯M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 _runtime‚ 188.359270938 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3525612 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4662426 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205497915904 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788416688621201e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14001361202230 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚879 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14161525236736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚860 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚18 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8188040 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚879 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15710660066304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 206153524736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4662426 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1151422 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13796637198336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14161525236736 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2055870 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14002032672006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3525614 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4643860 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚860 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚18 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14367005149696 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1150983 _runtime‚ 188.359270938 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8188040 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3525612 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3206851 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 168571904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4662426 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 205497915904 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 215862113280 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788416688621201e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3207292 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 205395473670 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14001361202230 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14161507233792 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 205497915904 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4643860 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15494990465024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1377329 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257668723200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5551458 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16922806137344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257839405568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 trainer/global_step‚299 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚890 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2456102 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5569998 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5551458 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 245414377906 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5551458 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17168470431232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 245414377906 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4212396 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18515876610560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17168470431232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1377329 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4212396 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 257668723200 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5551458 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16486939595776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16922806137344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456100 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 257839405568 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16486922556416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4212398 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5569998 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1376886 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9763854 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 trainer/global_step‚299B’Ž‚(:‡ Õ¥äÔÐÉ•~ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vâÊ:‚(:¯ Õ¥äÔèß“ disk.sda1.out‚0$ disk.nvme0n1.out‚794.13720703125 disk.sda2.out‚0) proc.memory.availableMB‚ 2546.83203125# proc.memory.rssMB‚ 2675.48046875) disk./.usagePercent‚16.90442581319462 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚794.13720703125 network.sent‚373146 disk.sda.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda3.out‚ 9.4921875 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚16.32983684539795 proc.cpu.threads‚17 disk.dm-1.out‚ 9.4921875 disk.rbd1.out‚ 0.1484375$ disk./.usageGB‚251.8881607055664 disk.drbd2080.out‚0 disk.dm-0.in‚0.09375! disk.dm-0.out‚794.13720703125 cpu‚25.078566038011314 network.recv‚166170 disk.sda.out‚ 9.4921875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375 disk.drbd2265.in‚0 disk.drbd2265.out‚0$ memory_percent‚84.45537090301514 disk.sda2.in‚0 disk.sda3.in‚0TL%-0+ _wandbruntime‚225  _runtime‚225FÕ4úµ&‘¯M E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5569912 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16787403069638 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788416725751239e+09 _step‚22 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17225642897920 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 225.489189551 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 trainer/global_step‚299 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892f´I¹&’³M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2464268 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258692281344 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5588478 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 164377600 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16979159176192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18577503408128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 246483721728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18836219629568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9814892 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258527903744 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 246230813078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16979177179136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5569912 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 245576382662 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16787403069638 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3845733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18836031311872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788416725751239e+09 _step‚22 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9796324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16788074539414 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17225642897920 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 225.489189551 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 246483721728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18577527348224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚16541826686976 trainer/global_step‚299 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ûQ®–0“+ _wandbruntime‚225  _runtime‚225ÇŽ‚O”É 1loss/SRqqt4/assignment_loss‚3.495374269263689 /loss/FBt3/assignment_loss‚0.376541049098191 1loss/FBt4/assignment_loss‚0.45988950158158937 _step‚23 /loss/FRt3/assignment_loss‚7.250670909881592 1loss/SRqqt2/assignment_loss‚3.397886481881142 1loss/FBt2/assignment_loss‚0.36834386128159063 .loss/FRt3/detection_loss‚4.072009962258744 0loss/SRqqt4/detection_loss‚3.960321545600891 /loss/FBt1/detection_loss‚21.120920673012733 &loss/total_loss‚113.01207620405474 epoch‚0 /loss/FRt2/assignment_loss‚7.320297339605907 0loss/FRt4/assignment_loss‚7.4299196067609286 2loss/SRqqt1/assignment_loss‚3.9050234174728393 0loss/SRqqt3/assignment_loss‚3.15232367515564 1loss/SRqqt1/detection_loss‚2.6727175789164357 1loss/SRqqt2/detection_loss‚0.8793820899793104 1loss/SRqqt3/detection_loss‚3.9740918636322022 % _timestamp‚1.7884167257528281e+09 _runtime‚ 225.490273877 /loss/FBt1/assignment_loss‚0.465683541377075 .loss/FRt1/detection_loss‚4.554999437332153 /loss/FBt2/detection_loss‚1.6305204769061177 .loss/FBt3/detection_loss‚7.218169320536696 trainer/global_step‚299 .loss/FRt1/assignment_loss‚7.32316370010376 /loss/FRt2/detection_loss‚2.4093545545188206 .loss/FRt4/detection_loss‚4.421752201883416 .loss/FBt4/detection_loss‚11.15271914601326ŒbŒ=S•Í .loss/FRt4/detection_loss‚4.421752201883416 .loss/FBt4/detection_loss‚11.15271914601326 1loss/SRqqt4/assignment_loss‚3.495374269263689 /loss/FBt3/assignment_loss‚0.376541049098191 1loss/FBt4/assignment_loss‚0.45988950158158937 _step‚23 /loss/FRt3/assignment_loss‚7.250670909881592 1loss/SRqqt2/assignment_loss‚3.397886481881142 1loss/FBt2/assignment_loss‚0.36834386128159063 .loss/FRt3/detection_loss‚4.072009962258744 0loss/SRqqt4/detection_loss‚3.960321545600891 /loss/FBt1/detection_loss‚21.120920673012733 &loss/total_loss‚113.01207620405474 epoch‚0 /loss/FRt2/assignment_loss‚7.320297339605907 0loss/FRt4/assignment_loss‚7.4299196067609286 2loss/SRqqt1/assignment_loss‚3.9050234174728393 0loss/SRqqt3/assignment_loss‚3.15232367515564 1loss/SRqqt1/detection_loss‚2.6727175789164357 1loss/SRqqt2/detection_loss‚0.8793820899793104 1loss/SRqqt3/detection_loss‚3.9740918636322022 % _timestamp‚1.7884167257528281e+09 _runtime‚ 225.490273877 /loss/FBt1/assignment_loss‚0.465683541377075 .loss/FRt1/detection_loss‚4.554999437332153 /loss/FBt2/detection_loss‚1.6305204769061177 .loss/FBt3/detection_loss‚7.218169320536696 trainer/global_step‚299 .loss/FRt1/assignment_loss‚7.32316370010376 /loss/FRt2/detection_loss‚2.4093545545188206QÆl’–‚(:‡ ä¥äÔ샘~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚184.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Et%ú<—‚(:± ä¥äÔ¯–à network.sent‚412139 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚251.87446212768555& disk.nvme0c0n1.out‚841.04541015625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2546.7421875! proc.memory.rssMB‚ 2675.484375 disk.rbd0.in‚0 disk.dm-1.out‚ 9.9609375 disk.drbd2266.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚0.09375 disk.dm-0.in‚0.09375! disk.dm-0.out‚841.04541015625 disk.rbd1.out‚ 0.1806640625 cpu‚25.076879728745297 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.out‚0 disk.sda3.out‚ 9.9609375 disk.drbd2080.in‚0 disk.sda.out‚ 9.9609375$ memory_percent‚84.45591926574707 network.recv‚172811 disk.rbd0.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚16.903506490139165 disk.drbd2265.in‚0 disk.drbd2266.out‚0$ disk.nvme0n1.out‚841.04541015625) proc.memory.percent‚16.32986068725586uÀ%e’˜‚(:‡ ó¥äÔíç¢~ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ðnò>™‚(:³ ó¥äÔҖ߀ disk.rbd1.in‚0$ memory_percent‚84.45477485656738 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.1806640625% disk./.usageGB‚251.88777923583984$ disk.nvme0n1.out‚888.76416015625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375 disk.dm-1.out‚ 10.8515625 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚888.76416015625 disk.sda2.in‚0 disk.sda3.out‚ 10.8515625 disk.nvme0n1.in‚0.09375 disk.sda.in‚0 disk.sda.out‚ 10.8515625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.32986068725586 network.sent‚451062* disk./.usagePercent‚16.904400212441196 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.09375 cpu‚25.07470314909853 network.recv‚178399 disk.drbd1716.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 2546.9296875! proc.memory.rssMB‚ 2675.484375 proc.cpu.threads‚17 disk.drbd2080.out‚0! disk.dm-0.out‚888.76416015625’êyÆ0š+ _wandbruntime‚261  _runtime‚261Ì®šËÄ›¾ & lr-AdamW/pg1‚4.869447954218544e-05 & lr-AdamW/pg2‚4.869447954218544e-05 trainer/global_step‚349 % _timestamp‚1.7884167616896796e+09 _runtime‚ 261.427198033 _step‚24Æ/EPȜ _runtime‚ 261.427198033 _step‚24 & lr-AdamW/pg1‚4.869447954218544e-05 & lr-AdamW/pg2‚4.869447954218544e-05 trainer/global_step‚349 % _timestamp‚1.7884167616896796e+09èÜg0+ _wandbruntime‚261  _runtime‚261 sîâ žªO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 287305695232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 286249717314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6477510 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 287305695232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚25 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19232129084416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19740476082688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 WKDeviÛ¤ÈÎceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4913198 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20027781777920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21899233738240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4913198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20027781777920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20027108179456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 286650099712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 rnÕ´'Ÿ®O ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 286249717314 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 286650099712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20027108179456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4913196 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6477510 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 287305695232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19740458079744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚25 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚19232129084416 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6496050 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19740476082688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2864500 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1608040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6496050 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4913198 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20027781777920 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21899233738240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4913198 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20027781777920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20027108179456 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 286650099712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4472069 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19518395841090 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19517724371522 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2864502 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11390706 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 287305695232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376Þ‡°0 + _wandbruntime‚262  _runtime‚262ûè0º&¡´M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19287033214976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4927214 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚ 262.129966357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884167623918123e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301674725888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6514530 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19287050254336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2872668 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19796811118592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 trainer/global_step‚349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19796829121536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30ÅUUq¢¸M SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚890 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19796829121536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19287050254336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6495964 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11441744 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1612605 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19573444937046 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20084954257920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2872668 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6514530 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19796811118592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2872670 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21961738957312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 287469527552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301508251136 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 286411722070 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 287469527552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11423176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4484810 trainer/global_step‚349 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19796829121536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19796811118592 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 288125136384 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19287033214976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4927214 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20084280646144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 287066152486 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚ 262.129966357 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚871 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4927212 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4485275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884167623918123e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21660040291328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 288125136384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6495964 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301674725888 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19574116406822 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1612142 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11441744 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11423176 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4927214 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6514530 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21961548542464qš 0£+ _wandbruntime‚262  _runtime‚262æ7&P¤Ê /loss/FRt2/assignment_loss‚7.203142781691118 /loss/FRt2/detection_loss‚2.7154571928761224 .loss/FBt3/detection_loss‚5.057735373731703 ,loss/FBt4/detection_loss‚9.1888730575641 trainer/global_step‚349 0loss/SRqqt3/assignment_loss‚3.42143746863964 1loss/SRqqt3/detection_loss‚3.6965935063916584 /loss/FBt2/detection_loss‚1.9247154160142004 _runtime‚ 262.131023853 % _timestamp‚1.7884167623935776e+09 /loss/FRt1/assignment_loss‚7.325473361545139 1loss/SRqqt1/assignment_loss‚3.829411641971485 2loss/SRqqt4/assignment_loss‚3.3020824526177077 1loss/FBt3/assignment_loss‚0.44486567141575506 1loss/FBt4/assignment_loss‚0.39691463482773137 .loss/FBt1/detection_loss‚7.815605433214278 .loss/FRt4/detection_loss‚3.746899237009612 0loss/SRqqt1/detection_loss‚2.636170949398843 1loss/SRqqt2/detection_loss‚1.9204677252411242 1loss/SRqqt4/detection_loss‚2.5568480589350715 _step‚27 1loss/SRqqt2/assignment_loss‚3.411223629064727 0loss/FBt1/assignment_loss‚0.5306157056973981 epoch‚0 /loss/FRt4/assignment_loss‚7.456301006403836 /loss/FRt3/detection_loss‚3.5868895914172754 /loss/FRt3/assignment_loss‚7.222024281819661 1loss/FBt2/assignment_loss‚0.42441627563146705 /loss/FRt1/detection_loss‚4.1217992846612574 %loss/total_loss‚93.93596373778091Eµ£T¥Î /loss/FBt2/detection_loss‚1.9247154160142004 _runtime‚ 262.131023853 % _timestamp‚1.7884167623935776e+09 /loss/FRt1/assignment_loss‚7.325473361545139 1loss/SRqqt1/assignment_loss‚3.829411641971485 2loss/SRqqt4/assignment_loss‚3.3020824526177077 1loss/FBt3/assignment_loss‚0.44486567141575506 1loss/FBt4/assignment_loss‚0.39691463482773137 .loss/FBt1/detection_loss‚7.815605433214278 .loss/FRt4/detection_loss‚3.746899237009612 0loss/SRqqt1/detection_loss‚2.636170949398843 1loss/SRqqt2/detection_loss‚1.9204677252411242 1loss/SRqqt4/detection_loss‚2.5568480589350715 _step‚27 1loss/SRqqt2/assignment_loss‚3.411223629064727 0loss/FBt1/assignment_loss‚0.5306157056973981 epoch‚0 /loss/FRt4/assignment_loss‚7.456301006403836 /loss/FRt3/detection_loss‚3.5868895914172754 /loss/FRt3/assignment_loss‚7.222024281819661 1loss/FBt2/assignment_loss‚0.42441627563146705 /loss/FRt1/detection_loss‚4.1217992846612574 %loss/total_loss‚93.93596373778091 /loss/FRt2/assignment_loss‚7.203142781691118 /loss/FRt2/detection_loss‚2.7154571928761224 .loss/FBt3/detection_loss‚5.057735373731703 ,loss/FBt4/detection_loss‚9.1888730575641 trainer/global_step‚349 0loss/SRqqt3/assignment_loss‚3.42143746863964 1loss/SRqqt3/detection_loss‚3.6965935063916584‘S6e¦‚(:„ ‚¦äÔþž±~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú*///§‚(:¤ ‚¦äÔ†…°€ disk.rbd1.out‚ 0.24365234375 disk.sda.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2546.76171875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda3.in‚0 disk.dm-1.out‚11.75 disk.rbd1.in‚0) proc.memory.percent‚16.32986068725586% disk./.usageGB‚251.86881637573242 disk.rbd0.out‚0% disk.nvme0c0n1.out‚936.1005859375 disk.sda3.out‚11.75 network.sent‚457484 network.recv‚183497 disk.sda.out‚11.75 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375 disk.drbd2265.out‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2675.484375* disk./.usagePercent‚16.903127598988487 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.09375 disk.dm-0.out‚936.1005859375 cpu‚25.073516496121336# disk.nvme0n1.out‚936.1005859375 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0$ memory_percent‚84.45580005645752 disk.dm-1.in‚0ßnV¨‚(:… ‘¦äÔ„Ŭ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.66' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04{œ»@©‚(:µ ‘¦äÔ¼ü‡ disk.sda3.in‚0 disk.sda3.out‚ 12.52734375 disk.drbd2265.in‚0 disk.dm-1.out‚ 12.52734375 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.09375# proc.memory.rssMB‚ 2675.48828125 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 12.52734375# disk.nvme0n1.out‚981.4521484375# memory_percent‚84.4559907913208 proc.cpu.threads‚17% disk.nvme0c0n1.out‚981.4521484375 cpu‚25.0722680050315 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.32988452911377* disk./.usagePercent‚16.905381745327475 disk.drbd2080.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2546.73046875 network.sent‚507488% disk./.usageGB‚251.90240478515625 disk.dm-0.out‚981.4521484375 disk.rbd1.out‚ 0.24365234375 disk.sda.in‚0 disk.sda2.in‚0 network.recv‚189585 disk.dm-0.in‚0.09375 disk.rbd1.in‚0:Xù0ª+ _wandbruntime‚298  _runtime‚298!\ºÄ«¾ % _timestamp‚1.7884167986690853e+09 _runtime‚ 298.406601355 _step‚28 & lr-AdamW/pg1‚5.567076600954725e-05 & lr-AdamW/pg2‚5.567076600954725e-05 trainer/global_step‚399t ÚÂȬ trainer/global_step‚399 % _timestamp‚1.7884167986690853e+09 _runtime‚ 298.406601355 _step‚28 & lr-AdamW/pg1‚5.567076600954725e-05 & lr-AdamW/pg2‚5.567076600954725e-05@n‘<0­+ _wandbruntime‚298  _runtime‚298öÊyÿ°'®ªO _step‚29 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5613996 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1839288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22885745927680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 298.407694018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25024753908736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7403562 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚24680950376960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5613998 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22886419526144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22558110022144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7422102 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚24680926436864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22558128025088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5112666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884167986698923e+09 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5613998 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7422102 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21977352651776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚21977335612416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22885745927680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22304437708498 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176973824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22303766238930 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22558110022144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚21977352651776 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13036100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13017558 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22885745927680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22558128025088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 327635905536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1839764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 298.407694018 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3272900 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25024753908736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 328291501056 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5112188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 328291501056 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚399 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7403562 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 327635905536 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 344004445696 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5613996—W0°+ _wandbruntime‚299  _runtime‚299o+Áña±´M ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚399 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 299.100561484 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 VHDeviceStatsMonitor.on_train_bat3@Y ch_end/requested_bytes.small_pool.current‚ 654430416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7440582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚30 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 329110942208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 327901491894 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13050028 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7422016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24742577174528 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚30 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5125390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13050028 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25087263636480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22360158274230 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5628012 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 329110942208 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5628014 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 327901491894 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22614463060992 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22943592006144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22614481063936 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7422016 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22032256782336 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22942918394368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13050028 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7422016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3281070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24742577174528 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13068596 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13068596 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5124932 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 344686461952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 trainer/global_step‚399 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 299.100561484 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22614481063936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22359486804454 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 344852936704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚24742601114624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25087454051328.ì0³+ _wandbruntime‚299  _runtime‚299Oý¹äR´Ì _step‚31 0loss/FBt1/assignment_loss‚0.2826393561618816 0loss/FBt4/assignment_loss‚0.7741432984670004 .loss/FRt2/detection_loss‚3.104386418327397 0loss/SRqqt3/detection_loss‚4.010259085972058 .loss/FBt3/detection_loss‚3.819864947526228 1loss/SRqqt3/assignment_loss‚3.369188390280071 2loss/SRqqt4/assignment_loss‚3.4183208245497485 0loss/FBt2/assignment_loss‚0.3501902565647684 /loss/FRt4/detection_loss‚3.1370964845021567 trainer/global_step‚399 1loss/SRqqt1/detection_loss‚2.3633331794697026 /loss/FBt4/detection_loss‚18.062645355860393 % _timestamp‚1.7884167993644361e+09 /loss/FRt3/assignment_loss‚7.256640911102295 1loss/SRqqt2/assignment_loss‚3.216317777633667 1loss/SRqqt2/detection_loss‚1.6363085733597296 0loss/SRqqt4/detection_loss‚2.319791453847518 /loss/FBt1/detection_loss‚18.575096047586865 &loss/total_loss‚109.53462093169009 epoch‚0 /loss/FRt1/assignment_loss‚7.394741370760161 .loss/FRt3/detection_loss‚2.660729666152124 1loss/FBt3/assignment_loss‚0.16325561558395357 /loss/FBt2/detection_loss‚2.0356106021161917 /loss/FRt2/assignment_loss‚7.288597995159673 /loss/FRt4/assignment_loss‚7.402627000621721 1loss/SRqqt1/assignment_loss‚3.306951173516207 /loss/FRt1/detection_loss‚3.5858851465685615 _runtime‚ 299.1018869062²qVµÐ 1loss/FBt3/assignment_loss‚0.16325561558395357 /loss/FBt2/detection_loss‚2.0356106021161917 /loss/FRt2/assignment_loss‚7.288597995159673 /loss/FRt4/assignment_loss‚7.402627000621721 1loss/SRqqt1/assignment_loss‚3.306951173516207 /loss/FRt1/detection_loss‚3.5858851465685615 _runtime‚ 299.101886906 _step‚31 0loss/FBt1/assignment_loss‚0.2826393561618816 0loss/FBt4/assignment_loss‚0.7741432984670004 .loss/FRt2/detection_loss‚3.104386418327397 0loss/SRqqt3/detection_loss‚4.010259085972058 .loss/FBt3/detection_loss‚3.819864947526228 1loss/SRqqt3/assignment_loss‚3.369188390280071 2loss/SRqqt4/assignment_loss‚3.4183208245497485 0loss/FBt2/assignment_loss‚0.3501902565647684 /loss/FRt4/detection_loss‚3.1370964845021567 trainer/global_step‚399 1loss/SRqqt1/detection_loss‚2.3633331794697026 /loss/FBt4/detection_loss‚18.062645355860393 % _timestamp‚1.7884167993644361e+09 /loss/FRt3/assignment_loss‚7.256640911102295 1loss/SRqqt2/assignment_loss‚3.216317777633667 1loss/SRqqt2/detection_loss‚1.6363085733597296 0loss/SRqqt4/detection_loss‚2.319791453847518 /loss/FBt1/detection_loss‚18.575096047586865 &loss/total_loss‚109.53462093169009 epoch‚0 /loss/FRt1/assignment_loss‚7.394741370760161 .loss/FRt3/detection_loss‚2.660729666152124Š”¶‘¶‚(:†  ¦äÔá±~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¬À¶B·‚(:·  ¦äÔˆõ˜€ disk.rbd1.in‚0 disk.sda3.out‚ 12.9609375$ memory_percent‚84.45611000061035 disk.drbd2266.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚514150 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 cpu‚25.071382820025427* disk./.usagePercent‚16.906948511437022 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.dm-1.in‚0( proc.memory.availableMB‚ 2546.7109375 disk.sda.out‚ 12.9609375 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 disk.drbd2080.in‚0 network.recv‚194952 disk.drbd1716.out‚0% disk.nvme0n1.out‚1029.03662109375 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚1029.03662109375 disk.drbd2265.in‚0 disk.dm-1.out‚ 12.9609375 disk.rbd1.out‚ 0.306640625 disk.sda2.out‚0 disk.dm-0.in‚0.09375) proc.memory.percent‚16.32990837097168% disk./.usageGB‚251.92575073242188" disk.dm-0.out‚1029.03662109375" proc.memory.rssMB‚ 2675.4921875Þæk/’¸‚(:‡ ¯¦äԇ̌~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.644' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3288 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̾R;¹‚(:° ¯¦äÔÒ¾¨€% disk.nvme0c0n1.out‚1068.861328125& proc.memory.availableMB‚ 2546.65625 disk.drbd1716.in‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0# memory_percent‚84.4564437866211) proc.memory.percent‚16.32990837097168 disk.sda1.out‚0 disk.sda3.in‚0* disk./.usagePercent‚16.907704501685636 disk.drbd2265.out‚0 cpu‚25.06969087214598 network.recv‚200230 disk.sda2.out‚0 disk.sda3.out‚ 13.69921875 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 disk.sda.out‚ 13.69921875 disk.drbd1716.out‚0# disk.nvme0n1.out‚1068.861328125 disk.drbd2265.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2675.4921875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 13.69921875 network.sent‚563413% disk./.usageGB‚251.93701553344727 disk.nvme0n1.in‚0.09375 disk.drbd2080.in‚0 disk.dm-0.out‚1068.861328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375¸ÿW’º‚(:‡ ¾¦äÔÙ¨˜~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚147.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.5862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}±Œ:»‚(:¯ ¾¦äÔ‘†¡€ proc.memory.rssMB‚2675.5$ disk./.usageGB‚251.9435577392578 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 network.sent‚569713 disk.dm-0.in‚0.09375 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0( proc.memory.percent‚16.3299560546875 disk.drbd2080.in‚0 disk.dm-1.out‚ 14.58984375 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.sda2.out‚0 disk.sda3.out‚ 14.58984375$ memory_percent‚84.45942401885986) proc.memory.availableMB‚ 2546.16796875& disk.nvme0c0n1.out‚1117.9970703125 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0$ disk.nvme0n1.out‚1117.9970703125 disk.rbd0.in‚0 disk.drbd2265.in‚0! disk.dm-0.out‚1117.9970703125 disk.rbd1.out‚ 0.306640625 cpu‚25.068870008500973 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 14.58984375 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚205166) disk./.usagePercent‚16.90814355460686¯µ0¼+ _wandbruntime‚335  _runtime‚335 ˆHĽ¾ & lr-AdamW/pg1‚6.264705247690906e-05 & lr-AdamW/pg2‚6.264705247690906e-05 trainer/global_step‚449 % _timestamp‚1.7884168352874956e+09 _runtime‚ 335.025009395 _step‚32©{ɾȾ & lr-AdamW/pg1‚6.264705247690906e-05 & lr-AdamW/pg2‚6.264705247690906e-05 trainer/global_step‚449 % _timestamp‚1.7884168352874956e+09 _runtime‚ 335.025009395 _step‚32 œ>µ0¿+ _wandbruntime‚335  _runtime‚3351£¯IÀªO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25375779967488 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 367920396130 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28150573232640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24722559179776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25744383675904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚ 335.026068595 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3681300 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884168352882092e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14644410 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚33 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27763463320064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 Ig atsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6314798 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14662952 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 386932067328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚25375779967488 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 367265965922 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 367920396130 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 368621711360 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28150573232640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8348154 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25744383675904 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24722559179776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 369277306880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚25745057274368 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚25745057274368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25375761964544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5752830 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25744383675904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚ 335.026068595 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3681300 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25375779967488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884168352882092e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14644410 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25090479575906 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚33 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27763463320064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24722542140416 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25432133006336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 370096748032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 335.711112025 _step‚34 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25802229754368 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚875 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 387788535296 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14695448 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5765957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25432115003392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25432133006336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25432133006336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14676880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24777463310336 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 370096748032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 369441139200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6328814 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2076030 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 370096748032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 335.711112025 _step‚34 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25146200141638 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5765498 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6328814 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8348068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25802229754368 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6328812 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 410263112704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28193413906944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 430126688768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7015596 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6392179 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27467765707776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9274206 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7015596 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16271262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28193431909888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31276126892032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 371.564589369 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9274206 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7015596 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27876521443314 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16271262 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28193431909888 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31276126892032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 371.564589369 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 410263112704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28193413906944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31276304737280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 430126688768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28603021424128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928%Ñ-þg0Ò+ _wandbruntime‚372  _runtime‚372·Ê×¹Ó²M TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9292686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28249766945792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788416872526183e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28249784948736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30907674881024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27522669838336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410426945024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2307525 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 408917740294 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 _step‚38 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27522652798976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28660193890816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28249766945792 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4097870 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9292686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31338648333312 _runtime‚ 372.26429869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 411082553856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28660867502592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚894 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7029614 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9292686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7029612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31338648333312 _runtime‚ 372.26429869 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16303732 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28660867502592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6404954 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9274120 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16322300 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27931570539270 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27932242009046 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 411082553856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16303732 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7029612 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 431155478528 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 430997392384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 411082553856 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28660867502592 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28249766945792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788416872526183e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28249784948736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2307086 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30907674881024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27522669838336 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 410426945024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2307525 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 408917740294 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16322300 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7029614 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 trainer/global_step‚499 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6405395 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 409572170710 _step‚38 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31338830359552&«à¢0Õ+ _wandbruntime‚372  _runtime‚372êÀï:GÖÁ $loss/total_loss‚92.3490517622767 /loss/FRt1/assignment_loss‚7.408799281487098 /loss/FRt2/assignment_loss‚7.268122735570689 1loss/SRqqt1/assignment_loss‚3.049211025238037 .loss/FRt3/detection_loss‚4.016707263432909 0loss/SRqqt1/detection_loss‚4.900153122629438 0loss/SRqqt3/detection_loss‚4.197380160963213 .loss/FBt4/detection_loss‚9.204918279367334 0loss/SRqqt3/assignment_loss‚2.97473406791687 1loss/SRqqt4/assignment_loss‚2.590911324535097 -loss/FRt4/detection_loss‚2.86956517842778 /loss/FRt3/assignment_loss‚7.278110671043396 .loss/FRt1/detection_loss‚4.327690419096213 % _timestamp‚1.7884168725279367e+09 /loss/FRt4/assignment_loss‚7.394573922742877 0loss/FBt1/assignment_loss‚0.2963964314665645 0loss/SRqqt4/detection_loss‚2.087268546170422 trainer/global_step‚499 /loss/FBt2/assignment_loss‚0.692724317514702 .loss/FBt2/detection_loss‚3.850108786448397 _runtime‚ 372.265386683 1loss/SRqqt2/assignment_loss‚2.606868364260747 /loss/FRt2/detection_loss‚2.5039325708248574 1loss/SRqqt2/detection_loss‚1.4353948083664796 .loss/FBt1/detection_loss‚8.045826336741447 _step‚39 1loss/FBt3/assignment_loss‚0.14768256646315217 0loss/FBt4/assignment_loss‚0.8461299490231527 .loss/FBt3/detection_loss‚2.355841632545838 epoch‚0<3R/K×Å .loss/FRt1/detection_loss‚4.327690419096213 % _timestamp‚1.7884168725279367e+09 /loss/FRt4/assignment_loss‚7.394573922742877 0loss/FBt1/assignment_loss‚0.2963964314665645 0loss/SRqqt4/detection_loss‚2.087268546170422 trainer/global_step‚499 /loss/FBt2/assignment_loss‚0.692724317514702 .loss/FBt2/detection_loss‚3.850108786448397 _runtime‚ 372.265386683 1loss/SRqqt2/assignment_loss‚2.606868364260747 /loss/FRt2/detection_loss‚2.5039325708248574 1loss/SRqqt2/detection_loss‚1.4353948083664796 .loss/FBt1/detection_loss‚8.045826336741447 _step‚39 1loss/FBt3/assignment_loss‚0.14768256646315217 0loss/FBt4/assignment_loss‚0.8461299490231527 .loss/FBt3/detection_loss‚2.355841632545838 epoch‚0 $loss/total_loss‚92.3490517622767 /loss/FRt1/assignment_loss‚7.408799281487098 /loss/FRt2/assignment_loss‚7.268122735570689 1loss/SRqqt1/assignment_loss‚3.049211025238037 .loss/FRt3/detection_loss‚4.016707263432909 0loss/SRqqt1/detection_loss‚4.900153122629438 0loss/SRqqt3/detection_loss‚4.197380160963213 .loss/FBt4/detection_loss‚9.204918279367334 0loss/SRqqt3/assignment_loss‚2.97473406791687 1loss/SRqqt4/assignment_loss‚2.590911324535097 -loss/FRt4/detection_loss‚2.86956517842778 /loss/FRt3/assignment_loss‚7.278110671043396'¹3ÏØ‚(:’ ë¦äÔŸ¾ˆ~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.011599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÙÛõCÙ‚(:¸ ë¦äÔ«àý network.recv‚219396! disk.dm-0.out‚1241.2900390625 disk.sda1.in‚0 disk.sda3.out‚ 16.20703125# proc.memory.rssMB‚ 2675.50390625 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2545.91796875 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0$ memory_percent‚84.46094989776611% disk./.usageGB‚251.99138259887695 disk.drbd2080.in‚0 disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375$ disk.nvme0n1.out‚1241.2900390625 network.sent‚631212 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 16.20703125 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.32997989654541 cpu‚25.065542839954674* disk./.usagePercent‚16.911353121063627 disk.sda.in‚0& disk.nvme0c0n1.out‚1241.2900390625 disk.sda2.out‚0 disk.dm-1.out‚ 16.20703125 proc.cpu.threads‚17Ë¡¤’Ú‚(:‡ ú¦äÔº¬~ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xŽ×AÛ‚(:¶ ú¦äÔš–„€ disk.sda2.out‚0 disk.drbd2265.in‚0$ memory_percent‚84.46018695831299) proc.memory.percent‚16.33000373840332 network.sent‚681244 network.recv‚225601) disk./.usagePercent‚16.91225093948621$ disk./.usageGB‚252.0047607421875 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0.09375" proc.memory.rssMB‚ 2675.5078125 disk.sda.out‚ 16.6953125 disk.sda3.out‚ 16.6953125 cpu‚25.065576675121243 proc.cpu.threads‚17" disk.dm-0.out‚1271.98974609375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚0.09375% disk.nvme0n1.out‚1271.98974609375 disk.sda3.in‚0) proc.memory.availableMB‚ 2546.04296875 disk.dm-0.in‚0.09375 disk.dm-1.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚1271.98974609375 disk.sda2.in‚0 disk.dm-1.out‚ 16.6953125 disk.rbd1.out‚ 0.306640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0c^I}’Ü‚(:‡ ‰§äÔ©ã’~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[_¤ïBÝ‚(:· ‰§äÔá©Á€ disk.sda.out‚ 17.53515625 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375& proc.memory.availableMB‚ 2546.53125 disk.drbd2265.out‚0 disk.sda2.out‚0* disk./.usagePercent‚16.912701512746473" disk.dm-0.out‚1311.25537109375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 network.sent‚689595 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.53515625 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375' disk.nvme0c0n1.out‚1311.25537109375 disk.dm-0.in‚0.09375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0$ memory_percent‚84.45720672607422) proc.memory.percent‚16.33002758026123# disk./.usageGB‚252.011474609375 disk.sda.in‚0 disk.sda3.out‚ 17.53515625 proc.cpu.threads‚17 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2675.51171875 network.recv‚230104 disk.rbd1.out‚ 0.306640625% disk.nvme0n1.out‚1311.25537109375 disk.sda3.in‚0 cpu‚25.064976805101065M,»0Þ+ _wandbruntime‚408  _runtime‚408 ×íGÄß¾ _runtime‚ 408.129815619 _step‚40 & lr-AdamW/pg1‚7.659962541163268e-05 & lr-AdamW/pg2‚7.659962541163268e-05 trainer/global_step‚549 % _timestamp‚1.7884169083922696e+09ÓÍË)Èà & lr-AdamW/pg1‚7.659962541163268e-05 & lr-AdamW/pg2‚7.659962541163268e-05 trainer/global_step‚549 % _timestamp‚1.7884169083922696e+09 _runtime‚ 408.129815619 _step‚40(¢™và0á+ _wandbruntime‚408  _runtime‚408R8­Ì´'â®O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚41 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30661891841154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚549 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 _runtime‚ 408.130771252 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884169083930144e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31011083852288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7716398 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31011065849344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17898114 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 473322938880 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31011065849344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414Ÿ‹}µåã²O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7716396 WKDeviceStatsMonitor.on_train_batch_start/inactive_GÛYÓ&split.small_pool.allocated‚2534824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚41 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚894 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30661891841154 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10181718 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31011083852288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7032461 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177845248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2534361 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚549 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 451248918528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 _runtime‚ 408.130771252 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31461659172352 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884169083930144e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31461659172352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34402037870592 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10200258 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7032926 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚875 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31011083852288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 451248918528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33928537086464 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30662563310722 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17916656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7716398 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7716396 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚30212955196416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4498100 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10200258 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31462332770816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31011065849344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17898114 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153905152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30212972235776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 473322938880 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 449591074946 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 450593323008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34401860025344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7716398 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33928561026560 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31011065849344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31462332770816 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 473476844032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31518831639040 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚42 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10218738 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4506268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 _runtime‚ 408.826307711 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7730414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884169090883164e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30718283876454 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31067418888192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚07¸Â&æ¼M C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚465 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153891840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30718283876454 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31067418888192 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7730412 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 452068359680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30717612406678 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17930584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34464384781824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4506270 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33990211764224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30267876366336 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31519505250816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚31518831639040 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17930584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚42 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17949152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 452068359680 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7045194 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30267859326976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7730412 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10218738 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4506268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2538926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33990187824128 _runtime‚ 408.826307711 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2539389 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7730414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34464562613760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31067436891136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31519505250816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884169090883164e+09 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 492234724352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37011073969664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8417196 _runtime‚ 445.140505751 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚599 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11126310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32958178763776 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34320970519040 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11107770 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11126310 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34320296920576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8417198 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4906502 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 492234724352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37011073969664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2766503 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33828717791744 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8417196 _runtime‚ 445.140505751 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37527889472000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 trainer/global_step‚599 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33828717791744 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 491579128832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19543508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11126310 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33828735794688 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32958178763776 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34320970519040 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 491579128832 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4906500 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 489771984146 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37011097909760 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚32958161724416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11107770 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 492234724352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34320296920576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19524966 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11126310 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚33828735794688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34320296920576 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37528058928640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8417198 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7673003 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7686247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11126224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 517853663744 _step‚46 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490588419110 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33013065854976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33885088833536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37072748647424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 _runtime‚ 445.826946869 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34378142999040 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2771105 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.largÅæûÒe_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 trainer/global_step‚599 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884169460886116e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 8682209283Þç>Â&ö¼M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19576004 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8431214 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19576004 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19557436 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33013082894336 trainer/global_step‚599 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884169460886116e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34378142999040 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19557436 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7685773 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚33503654274086 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11144790 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7686247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2771577 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37590426576384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚33504325743862 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11126224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37590602311168 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 517853663744 _step‚46 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 490588419110 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8431212 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11126224 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 492398556672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚474 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4914670 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33013065854976 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33885088833536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37072748647424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11144790 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37072724707328 _runtime‚ 445.826946869 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34377469387264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34378142999040 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2771105 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8431214 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 trainer/global_step‚649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 560095342080 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 % _timestamp‚1.7884169820381136e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9117998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 532564934656 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 531261753762 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36646369734144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559928854016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 530607323554 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 _runtime‚ 481.775817928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40653730135040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872¹©@&…²O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚49 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 166488064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36646387737088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 533220530176 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12033822 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5314900 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12052362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37178934668800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 533220530176 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 trainer/global_step‚649 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 560095342080 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚36234647045538 % _timestamp‚1.7884169820381136e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36646369734144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36233975575970 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9117998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37179608267264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12033822 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40093634792960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peakV…ªÓ©‚22630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 532564934656 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9117996 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21170360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12052362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 531261753762 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2997161 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40653539706880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37179608267264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36646369734144 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9117998 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559928854016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 530607323554 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8312061 _runtime‚ 481.775817928 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 532564934656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40653730135040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36646387737088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37178934668800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35703385291776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37236780747264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12070842 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40155261590528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5323070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8325307 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 533384362496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40155285530624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35758272382976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884169827244115e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40716073562112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 482.462419115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9132012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚50 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36702722772992 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9132014 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36702722772992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928WËKÂ&ˆ¼M 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 482.462419115 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9132012 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚50 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37236780747264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21184288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36702722772992 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 533384362496 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9132014 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚40716251394048 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 560811971584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21184288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 531423758518 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36702722772992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12052276 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 532078188934 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36702740775936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5323068 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177831936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚36289696141494 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12052276 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37236780747264 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12070842 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12070842 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40155261590528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5323070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21202856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36290367611270 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8325307 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9132014 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9132012 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 533384362496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40155285530624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8324857 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35758272382976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 560965863424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884169827244115e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3002237 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3001789 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚407160735621122RN·0‰+ _wandbruntime‚482  _runtime‚482—|?#NŠÈ /loss/FBt4/detection_loss‚14.088714373111724 %loss/total_loss‚93.14740192640988 .loss/FRt1/detection_loss‚4.356139703875496 .loss/FRt2/detection_loss‚3.704421927289265 1loss/SRqqt1/assignment_loss‚3.085094405072076 1loss/SRqqt3/assignment_loss‚2.681114302231715 1loss/FBt3/assignment_loss‚0.03232802963256373 -loss/FRt4/detection_loss‚2.72504812824939 % _timestamp‚1.7884169827259614e+09 1loss/SRqqt2/assignment_loss‚2.874757584184408 0loss/SRqqt1/detection_loss‚2.725504635082507 _step‚51 2loss/SRqqt4/assignment_loss‚2.7132752442901786 1loss/SRqqt2/detection_loss‚1.6118713541278762 0loss/FBt1/assignment_loss‚0.4211503586568422 0loss/FBt2/assignment_loss‚0.6305018345083226 0loss/SRqqt3/detection_loss‚2.768458200045503 .loss/FBt2/detection_loss‚6.534704972058535 /loss/FBt3/detection_loss‚2.5341501721642006 0loss/FBt4/assignment_loss‚0.7850739523768425 /loss/FRt2/assignment_loss‚7.235244274139404 /loss/FRt3/assignment_loss‚7.169444195059842 /loss/FBt1/detection_loss‚5.8143264307425575 /loss/FRt4/assignment_loss‚6.452405605997358 epoch‚0 /loss/FRt1/assignment_loss‚7.364175773802257 /loss/FRt3/detection_loss‚3.3823041567585004 trainer/global_step‚649 _runtime‚ 482.463678061 0loss/SRqqt4/detection_loss‚1.457192312952512ǶJ;R‹Ì epoch‚0 /loss/FRt1/assignment_loss‚7.364175773802257 /loss/FRt3/detection_loss‚3.3823041567585004 trainer/global_step‚649 _runtime‚ 482.463678061 0loss/SRqqt4/detection_loss‚1.457192312952512 /loss/FBt4/detection_loss‚14.088714373111724 %loss/total_loss‚93.14740192640988 .loss/FRt1/detection_loss‚4.356139703875496 .loss/FRt2/detection_loss‚3.704421927289265 1loss/SRqqt1/assignment_loss‚3.085094405072076 1loss/SRqqt3/assignment_loss‚2.681114302231715 1loss/FBt3/assignment_loss‚0.03232802963256373 -loss/FRt4/detection_loss‚2.72504812824939 % _timestamp‚1.7884169827259614e+09 1loss/SRqqt2/assignment_loss‚2.874757584184408 0loss/SRqqt1/detection_loss‚2.725504635082507 _step‚51 2loss/SRqqt4/assignment_loss‚2.7132752442901786 1loss/SRqqt2/detection_loss‚1.6118713541278762 0loss/FBt1/assignment_loss‚0.4211503586568422 0loss/FBt2/assignment_loss‚0.6305018345083226 0loss/SRqqt3/detection_loss‚2.768458200045503 .loss/FBt2/detection_loss‚6.534704972058535 /loss/FBt3/detection_loss‚2.5341501721642006 0loss/FBt4/assignment_loss‚0.7850739523768425 /loss/FRt2/assignment_loss‚7.235244274139404 /loss/FRt3/assignment_loss‚7.169444195059842 /loss/FBt1/detection_loss‚5.8143264307425575 /loss/FRt4/assignment_loss‚6.4524056059973583ÝS1’Œ‚(:‡ ã§äÔôà¡~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08\—³A‚(:¶ ã§äÔ—䬀 disk.drbd2080.out‚0 disk.dm-0.in‚0.09375* disk./.usagePercent‚16.916477111861447 disk.nvme0n1.in‚0.09375 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 21.98046875 network.sent‚811843 network.recv‚260989 disk.sda.in‚0 disk.sda.out‚ 21.98046875 disk.sda1.out‚0 disk.dm-1.out‚ 21.98046875% disk./.usageGB‚252.06773376464844 disk.nvme0c0n1.in‚0.09375 disk.sda3.in‚0! disk.dm-0.out‚1597.6259765625 disk.dm-1.in‚0$ memory_percent‚84.45773124694824! proc.memory.rssMB‚ 2675.515625 proc.cpu.threads‚17 disk.sda1.in‚0& disk.nvme0c0n1.out‚1597.6259765625 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 2546.4453125) proc.memory.percent‚16.33005142211914 cpu‚25.061447321549505 disk.drbd1716.in‚0$ disk.nvme0n1.out‚1597.6259765625 disk.rbd0.out‚0 disk.rbd1.out‚ 0.306640625 disk.drbd1716.out‚0 disk.drbd2265.out‚0ü_M’Ž‚(:‡ ò§äԸʉ~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.609' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÕ¤_?‚(:´ ò§äÔ†×ñ€ network.sent‚861828 disk.drbd2265.in‚0 disk.sda.out‚ 22.61328125 disk.sda1.in‚0$ disk.nvme0n1.out‚1638.9853515625$ memory_percent‚84.45901870727539' proc.memory.availableMB‚ 2546.234375 disk.rbd1.out‚ 0.306640625% disk./.usageGB‚252.07619094848633 disk.nvme0c0n1.in‚0.09375& disk.nvme0c0n1.out‚1638.9853515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0! disk.dm-0.out‚1638.9853515625 disk.dm-1.out‚ 22.61328125 disk.rbd1.in‚0 disk.drbd1716.out‚0) proc.memory.percent‚16.33005142211914 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 22.61328125 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 network.recv‚267255* disk./.usagePercent‚16.917044680564857 disk.sda3.in‚0 disk.nvme0n1.in‚0.09375! proc.memory.rssMB‚ 2675.515625 cpu‚25.06107406359988ºêž(0+ _wandbruntime‚518  _runtime‚518’žÑR‘¼ & lr-AdamW/pg1‚9.752848481371811e-05 & lr-AdamW/pg2‚9.752848481371811e-05 trainer/global_step‚699 $ _timestamp‚1.788417018597025e+09 _runtime‚ 518.33453119 _step‚52˜xÆ’À _runtime‚ 518.33453119 _step‚52 & lr-AdamW/pg1‚9.752848481371811e-05 & lr-AdamW/pg2‚9.752848481371811e-05 trainer/global_step‚699 $ _timestamp‚1.788417018597025e+094÷“0“+ _wandbruntime‚518  _runtime‚518‚Ö=Û”®O TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_kÝ)Ù&bytes.all.current‚ 671469568 _step‚53 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8954099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12959874 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9818798Åü¸'•²O SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5723300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43176171676160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22778670 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3230334 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 573550740480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9818798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12978414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _step‚53 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8954099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3230797 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40038246015488 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12959874 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40038246015488 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40037572417024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39020688912946 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 574206336000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 572097093170 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40037572417024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12959874 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39020017443378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 8630487045kg$0–+ _wandbruntime‚519  _runtime‚519H%ã¼¾&—¸M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9832812 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39520392718336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3235357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 _step‚54 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604150903296 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43237798473728 trainer/global_step‚699 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43237798473728 trainer/global_step‚699 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39076409478678 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604317378048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8966350 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39075738008902 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 572913528342 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_train_batchÅq²d_end/allocated_bytes.small_pool.current‚ 655608832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12996894 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43841949377024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9832812 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 572259097926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8966827 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40095418495488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22811140 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39520392718336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 575025777152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3235357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22829708 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22829708 _step‚54 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39520392718336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 574370168320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39520374715392 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3234882 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38503495950336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5731468 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚43842139791872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22811140 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43237822413824 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6041509032966»(0™+ _wandbruntime‚519  _runtime‚519 n}PšÊ 1loss/FBt3/assignment_loss‚0.37542728361513583 1loss/SRqqt2/detection_loss‚1.8496195556737949 %loss/total_loss‚82.05194941127243 % _timestamp‚1.7884170192705002e+09 2loss/SRqqt2/assignment_loss‚3.1058452013062268 2loss/SRqqt3/assignment_loss‚2.7574737131595612 0loss/FBt1/assignment_loss‚0.6097644129040418 0loss/FBt2/assignment_loss‚0.5774769847979769 .loss/FRt4/detection_loss‚2.027056833234014 /loss/FRt2/detection_loss‚3.1864463255113487 0loss/SRqqt3/detection_loss‚3.573403898626566 /loss/SRqqt1/detection_loss‚3.52883134214651 /loss/FBt3/detection_loss‚3.4192782693374153 0loss/FBt4/assignment_loss‚0.4276479254622245 /loss/FRt2/assignment_loss‚7.351944767698949 0loss/FRt3/assignment_loss‚7.2238497575124105 0loss/SRqqt1/assignment_loss‚2.62858316728047 .loss/FRt1/detection_loss‚4.110257370604409 .loss/FBt4/detection_loss‚8.804731962271035 _runtime‚ 519.007948096 .loss/FRt3/detection_loss‚3.794865255160465 1loss/SRqqt4/detection_loss‚1.6786574014323823 /loss/FBt1/detection_loss‚3.9307437475791254 epoch‚0 /loss/FRt4/assignment_loss‚5.095939047728913 1loss/SRqqt4/assignment_loss‚2.580479256016143 /loss/FBt2/detection_loss‚2.2422986326714023 trainer/global_step‚699 _step‚55 /loss/FRt1/assignment_loss‚7.171327299541897}_vT›Î /loss/FBt3/detection_loss‚3.4192782693374153 0loss/FBt4/assignment_loss‚0.4276479254622245 /loss/FRt2/assignment_loss‚7.351944767698949 0loss/FRt3/assignment_loss‚7.2238497575124105 0loss/SRqqt1/assignment_loss‚2.62858316728047 .loss/FRt1/detection_loss‚4.110257370604409 .loss/FBt4/detection_loss‚8.804731962271035 _runtime‚ 519.007948096 .loss/FRt3/detection_loss‚3.794865255160465 1loss/SRqqt4/detection_loss‚1.6786574014323823 /loss/FBt1/detection_loss‚3.9307437475791254 epoch‚0 /loss/FRt4/assignment_loss‚5.095939047728913 1loss/SRqqt4/assignment_loss‚2.580479256016143 /loss/FBt2/detection_loss‚2.2422986326714023 trainer/global_step‚699 _step‚55 /loss/FRt1/assignment_loss‚7.171327299541897 1loss/FBt3/assignment_loss‚0.37542728361513583 1loss/SRqqt2/detection_loss‚1.8496195556737949 %loss/total_loss‚82.05194941127243 % _timestamp‚1.7884170192705002e+09 2loss/SRqqt2/assignment_loss‚3.1058452013062268 2loss/SRqqt3/assignment_loss‚2.7574737131595612 0loss/FBt1/assignment_loss‚0.6097644129040418 0loss/FBt2/assignment_loss‚0.5774769847979769 .loss/FRt4/detection_loss‚2.027056833234014 /loss/FRt2/detection_loss‚3.1864463255113487 0loss/SRqqt3/detection_loss‚3.573403898626566 /loss/SRqqt1/detection_loss‚3.528831342146517pM‘œ‚(:† ¨äÔºÙ×~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0waX?‚(:´ ¨äÔŽþ‚€ disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375 disk.sda.out‚ 23.13671875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 0.306640625 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0$ memory_percent‚84.45959091186523! proc.memory.rssMB‚ 2675.515625) disk./.usagePercent‚16.91752136659361 disk.dm-1.out‚ 23.13671875 disk.drbd2266.in‚0 disk.sda2.out‚0 cpu‚25.060855064150996 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2265.in‚0$ disk.nvme0n1.out‚1675.2900390625 network.sent‚868056 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚271512 disk.sda3.in‚0 disk.drbd2080.in‚0! disk.dm-0.out‚1675.2900390625% disk./.usageGB‚252.08329391479492 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0' proc.memory.availableMB‚ 2546.140625) proc.memory.percent‚16.33005142211914 disk.nvme0c0n1.in‚0.09375& disk.nvme0c0n1.out‚1675.2900390625 disk.sda3.out‚ 23.13671875sB¦³ž‚(:’ ¨äÔÌ—©~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.895799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fCc]>Ÿ‚(:³ ¨äÔѹº€% disk./.usageGB‚252.09634017944336 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.nvme0n1.out‚1720.626953125 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda3.out‚ 23.70703125 disk.dm-0.out‚1720.626953125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0) proc.memory.percent‚16.33005142211914 cpu‚25.060527569414077 disk.nvme0n1.in‚0.09375 disk.dm-0.in‚0.09375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.306640625! proc.memory.rssMB‚ 2675.515625* disk./.usagePercent‚16.918396912360713 disk.sda1.out‚0 disk.drbd1716.out‚0 network.sent‚918491 disk.rbd0.in‚0 disk.dm-1.out‚ 23.70703125( proc.memory.availableMB‚ 2546.1328125 network.recv‚278370% disk.nvme0c0n1.out‚1720.626953125 disk.sda.out‚ 23.70703125$ memory_percent‚84.45963859558105 disk.sda3.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17‰qWì’ ‚(:‡ Ÿ¨äÔËÿ~ gpu.0.gpu‚18.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0מ:¡‚(:¯ Ÿ¨äÔìŦ€ disk.sda.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375 disk.sda3.out‚ 24.2734375) proc.memory.availableMB‚ 2546.12109375 network.sent‚924107( disk./.usagePercent‚16.9194465432511 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2675.51953125 cpu‚25.0602697364606 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.dm-0.out‚1759.470703125 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0) proc.memory.percent‚16.33007526397705 disk.drbd2080.out‚0 disk.dm-1.out‚ 24.2734375 disk.drbd1716.in‚0 disk.sda2.in‚0$ memory_percent‚84.45971012115479 network.recv‚282225 disk.sda.out‚ 24.2734375 disk.nvme0c0n1.in‚0.09375% disk.nvme0c0n1.out‚1759.470703125% disk./.usageGB‚252.11198043823242 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚1759.470703125 disk.dm-0.in‚0.09375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17Åú0¢+ _wandbruntime‚555  _runtime‚555½û9ãÅ£¿ $ _timestamp‚1.788417055335525e+09 _runtime‚ 555.073046414 _step‚56 ' lr-AdamW/pg1‚0.00010450477128107991 ' lr-AdamW/pg2‚0.00010450477128107991 trainer/global_step‚749yº´»É¤Ã trainer/global_step‚749 $ _timestamp‚1.788417055335525e+09 _runtime‚ 555.073046414 _step‚56 ' lr-AdamW/pg1‚0.00010450477128107991 ' lr-AdamW/pg2‚0.000104504771281079918êó?0¥+ _wandbruntime‚555  _runtime‚555 AfÑ·'¦±O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46905115699200 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24405522 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10519598 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 646431079936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46905295641600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚57 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42281691621888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42896210165248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973M`õ<§µO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42281673618944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9594046 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24405522 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10519598 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42281673618944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42281691621888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46258684619264 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 156002304 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41193781308416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6131700 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42896210165248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 646431079936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46258708559360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13904466 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46905295641600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24424064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9594514 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚57 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42281691621888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 615192141824 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42896210165248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13885926 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 612932432578 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42896883763712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13904466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46905115699200 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 614536546304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42896883763712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179942400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 101942558729ð„ÁÕ0¨+ _wandbruntime‚555  _runtime‚555L›87Â&©¼M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 616011582976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42953382631936 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42338044660736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 613748867750 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42954056243712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚58 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9607350 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42338044660736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41248702478336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46967630907392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 555.768790358 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884170560306938e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46967815030784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10533612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13904380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚58 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9607350 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41248685438976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24456560 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10533614 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42338044660736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42338026657792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10533614 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13922946 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9606872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 184123392 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 647455733760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41248702478336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 160183296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46967630907392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 555.768790358 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3467480 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884170560306938e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 615355974144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46967815030784 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46320359297024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24456560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24437992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10533612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13904380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41862451346086 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3467004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 613748867750 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6139870 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 trainer/global_step‚749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42954056243712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 615355974144 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11220396 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45754847913472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚CßM<Ó414 _step‚61 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14830518 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50030793241088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11220398 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11220398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 656177947648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 158099456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0#(ý'··O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45099325561344 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44592101178194 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 689729838080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14811978 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44592772647762 trainer/global_step‚799 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 655522352128 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6540102 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3694379 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 689571738624 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43938987836416 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 655522352128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 _runtime‚ 591.452544229 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 656177947648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3694845 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43939004875776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45754847913472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11220396 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45754847913472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45099343564288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚61 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 653113341778 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45099325561344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45755521511936 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26050916 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49341245442560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49341221502464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14830518 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50030793241088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26032374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11220398 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11220398 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 653767771986 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45755521511936 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 656177947648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 182039552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 158099456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45099343564288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568=êA=_0¸+ _wandbruntime‚592  _runtime‚592ê¹ÞÄ&¹¾M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26064844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3698982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49402896180224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚62 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45155678600192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 656997388800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 % _timestamp‚1.7884170924014764e+09 _runtime‚ 592.139585486 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 trainer/global_step‚799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26083412 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50093499616768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚62 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45155678600192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 656997388800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10247250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43993891966976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 % _timestamp‚1.7884170924014764e+09 _runtime‚ 592.139585486 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 trainer/global_step‚799 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11234414 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44648493213494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45155696603136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155988992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44647821743718 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45155696603136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11234412 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6548268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45155678600192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6548270 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3699443 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3698982 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49402896180224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BE\¶ä0»+ _wandbruntime‚592  _runtime‚592u“t$P¼Ê /loss/SRqqt3/detection_loss‚4.27669377438724 0loss/FBt2/assignment_loss‚0.7839247118681669 /loss/FBt1/detection_loss‚5.2658501895437855 trainer/global_step‚799 .loss/FRt1/detection_loss‚3.269886182702106 0loss/SRqqt2/detection_loss‚4.030238867737353 _step‚63 /loss/FRt3/assignment_loss‚5.875592347347375 /loss/FRt4/assignment_loss‚4.499739033835275 2loss/SRqqt4/assignment_loss‚2.3701922761655485 .loss/FRt2/detection_loss‚4.102325884004434 /loss/FBt3/detection_loss‚3.4399371957293385 0loss/FBt1/assignment_loss‚0.6923557457026058 1loss/FBt4/assignment_loss‚0.29467177460641586 /loss/FRt2/assignment_loss‚6.887401117218865 2loss/SRqqt3/assignment_loss‚2.3413018537685275 1loss/FBt3/assignment_loss‚0.43158615284260626 1loss/SRqqt4/detection_loss‚1.0278917193676518 %loss/total_loss‚82.23643272489144 epoch‚0 /loss/FRt1/assignment_loss‚6.666095920231031 2loss/SRqqt2/assignment_loss‚3.1263183653354645 .loss/FBt2/detection_loss‚6.057919297899518 .loss/FBt4/detection_loss‚3.748637093816485 $ _timestamp‚1.788417092403486e+09 _runtime‚ 592.140938329 2loss/SRqqt1/assignment_loss‚2.6937335173288983 .loss/FRt3/detection_loss‚4.208605041067378 /loss/FRt4/detection_loss‚1.5599978009656983 0loss/SRqqt1/detection_loss‚4.585536861419678xC2T½Î 0loss/FBt2/assignment_loss‚0.7839247118681669 /loss/FBt1/detection_loss‚5.2658501895437855 trainer/global_step‚799 .loss/FRt1/detection_loss‚3.269886182702106 0loss/SRqqt2/detection_loss‚4.030238867737353 _step‚63 /loss/FRt3/assignment_loss‚5.875592347347375 /loss/FRt4/assignment_loss‚4.499739033835275 2loss/SRqqt4/assignment_loss‚2.3701922761655485 .loss/FRt2/detection_loss‚4.102325884004434 /loss/FBt3/detection_loss‚3.4399371957293385 0loss/FBt1/assignment_loss‚0.6923557457026058 1loss/FBt4/assignment_loss‚0.29467177460641586 /loss/FRt2/assignment_loss‚6.887401117218865 2loss/SRqqt3/assignment_loss‚2.3413018537685275 1loss/FBt3/assignment_loss‚0.43158615284260626 1loss/SRqqt4/detection_loss‚1.0278917193676518 %loss/total_loss‚82.23643272489144 epoch‚0 /loss/FRt1/assignment_loss‚6.666095920231031 2loss/SRqqt2/assignment_loss‚3.1263183653354645 .loss/FBt2/detection_loss‚6.057919297899518 .loss/FBt4/detection_loss‚3.748637093816485 $ _timestamp‚1.788417092403486e+09 _runtime‚ 592.140938329 2loss/SRqqt1/assignment_loss‚2.6937335173288983 .loss/FRt3/detection_loss‚4.208605041067378 /loss/FRt4/detection_loss‚1.5599978009656983 0loss/SRqqt1/detection_loss‚4.585536861419678 /loss/SRqqt3/detection_loss‚4.27669377438724?™Eûi’¾‚(:‡ ̨äÔ«÷´~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PéÚÖE¿‚(:º ̨äԦꭂ% disk./.usageGB‚252.12176132202148 disk.drbd2265.out‚0 disk.dm-1.out‚ 25.66015625 disk.rbd1.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚1868.33740234375 disk.rbd0.in‚0 network.sent‚987608* disk./.usagePercent‚16.920102946568893 disk.drbd2080.out‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚1868.33740234375 disk.nvme0c0n1.in‚0.09375" proc.memory.rssMB‚ 2675.5234375 network.recv‚298815 disk.sda3.out‚ 25.66015625 disk.drbd2080.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda.in‚0 disk.sda.out‚ 25.66015625) proc.memory.availableMB‚ 2545.98046875 disk.drbd1716.out‚0$ memory_percent‚84.46056842803955 cpu‚25.05936106162486 disk.drbd2265.in‚0" disk.dm-0.out‚1868.33740234375 disk.drbd2266.out‚0 disk.nvme0n1.in‚0.09375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0) proc.memory.percent‚16.33009910583496 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0Ô/È’À‚(:‡ Û¨äÔ¦±ž~ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.846' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07fâÙ?Á‚(:´ Û¨äÔ𤤀 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375% disk.nvme0n1.out‚1912.22802734375 disk.drbd2266.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚0.09375( proc.memory.availableMB‚ 2545.9765625 disk.sda.in‚0' disk.nvme0c0n1.out‚1912.22802734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚1912.22802734375 disk.drbd2265.in‚0 network.sent‚ 1.037127e+06 network.recv‚304542 disk.dm-1.in‚0 disk.rbd1.out‚ 0.306640625 disk.nvme0c0n1.in‚0.09375$ memory_percent‚84.46059226989746) proc.memory.percent‚16.33009910583496* disk./.usagePercent‚16.921738066690086% disk./.usageGB‚252.14612579345703 disk.sda.out‚26.1875 disk.drbd1716.in‚0 cpu‚25.058920144746416 disk.dm-1.out‚26.1875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda3.out‚26.1875" proc.memory.rssMB‚ 2675.5234375‚Ó¢0Â+ _wandbruntime‚628  _runtime‚628ßóÅÿ $ _timestamp‚1.788417128651122e+09 _runtime‚ 628.388665054 _step‚64 ' lr-AdamW/pg1‚0.00011845734421580353 ' lr-AdamW/pg2‚0.00011845734421580353 trainer/global_step‚849ҲЖÉÄà ' lr-AdamW/pg1‚0.00011845734421580353 ' lr-AdamW/pg2‚0.00011845734421580353 trainer/global_step‚849 $ _timestamp‚1.788417128651122e+09 _runtime‚ 628.388665054 _step‚64@øåg0Å+ _wandbruntime‚628  _runtime‚628æ¶/^º'Æ´O G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52423758385664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15738030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27677768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15738030 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6948502 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 732867627008 % _timestamp‚1.7884171286518323e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15756570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 697163753472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3927876 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11921196 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48613485661696 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 697163753472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52423782325760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 145516544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3927876 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11921196 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47378143045602 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48614159260160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 169456640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15738030 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46684211403776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47378814515170 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27677768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15756570 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27659226 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15738030 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3927446 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694603111394 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693948681186 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10876378 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6948502 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 732867627008 % _timestamp‚1.7884171286518323e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15756570 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 I-0È+ _wandbruntime‚629  _runtime‚629pz{¶ ɽM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3932516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47973348545536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 697327585792 OADeviceStatsMonitor.on_train_batch_end/allocated=-¥É´_bytes.all.current‚ 673611776 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733897130496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 629.083318619 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 _step‚66 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚849 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47433863611126 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6956670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 694765116150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3932516 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11935212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11935214 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47973348545536 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53219330193920 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 733734850048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47434535080902 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11935214 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46739098494976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27691696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 697327585792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 733897130496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 629.083318619 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27710264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53219143973376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48671331740160 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10888735 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 _step‚66 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚885 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52485433063424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48670658128384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚849 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15756484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47433863611126 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695419546566 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11935212 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6956670 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27710264 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27691696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 738149559296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51472797008384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50734629446144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50164856382578 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 738149559296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11516701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7356902 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.5<ï peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 trainer/global_step‚899 _runtime‚ 664.981076332 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16682622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚69 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49429400892416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55506295268864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12621998 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 162293760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49429417931776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12621998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚478 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 735438450802 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 738149559296 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51472797008384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚50734629446144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 776253082624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16664082 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51472123409920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 186233856 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16682622 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50164856382578 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 738149559296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29304620 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11516701 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7356902 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 734784020594 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12621996 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 trainer/global_step‚899 _runtime‚ 664.981076332 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16682622 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56282572291584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16664082 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56282386057728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚69 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12621996 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚49429400892416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55506295268864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 % _timestamp‚1.7884171659267335e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51529969488384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7365070 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 738969000448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7365068 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 738313391616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 735600455558 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50219905478534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50790982484992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49484305022976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4164924 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 665.6649317 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50791000487936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0$I¿´>&ÜÀM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50791000487936 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50791000487936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚70 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 738969000448 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4164457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12636014 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55567946006528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 736254885974 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56344916966912 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚50790982484992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51529969488384 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 % _timestamp‚1.7884171659267335e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 738313391616 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51529969488384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16701102 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 776970960384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29337116 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚55567969946624 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚899 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7365070 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 738969000448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151794688 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318548 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 777122755072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29337116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50220576948310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7365068 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11529994 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11529525 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 738313391616 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12636014 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 735600455558 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12636012 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50219905478534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12636012 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50790982484992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚49484305022976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4164924 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 665.6649317 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56345092701696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 175734784 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779135365120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13322798 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 819394773504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778479769600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53552299391488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12156984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30931472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 819221993984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 trainer/global_step‚949 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30912930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53552281388544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414žÛi¦½'é·O _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚52174624459776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚53552299391488 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12156984 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7765300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13322798 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13322796 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54330761158144 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779135365120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30931472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12156485 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 819221993984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59408250865664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59408054146048 trainer/global_step‚949 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30912930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚54331434756608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196719616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚53552281388544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4391185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778479769600 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52950898249986 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚ 701.907534951 _step‚73 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58588856092160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 $ _timestamp‚1.788417202169791e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7765302 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4391682 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58588832152064 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52950226780418 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172779520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17608674 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚54331434756608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17590134 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54330761158144 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779135365120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13322796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52174607420416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17590134 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13322798 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17608674 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 819394773504 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778479769600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872IÏÙBæ0ê+ _wandbruntime‚702  _runtime‚702~Naöë¾M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17608588 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884172028764594e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779299197440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚949 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54388607236608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13336814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 96436914f5M0Î77 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7773468 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17608588 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53608652430336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928ð\ß_È&ìÂM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59470574795776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 779954806272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17608588 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30945400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54388607236608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884172028764594e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30945400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30963968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30963968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58650506829824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12169198 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4395730 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 779954806272 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779299197440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59470756822016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚949 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54388607236608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13336812 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52229528590336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777090225382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 820091906048 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 776435794966 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13336814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52229511550976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58650482889728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚479 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17627154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7773468 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17608588 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53006618815718 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53608652430336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Jµï\#0í+ _wandbruntime‚702  _runtime‚702oð®ÚPîÊ % _timestamp‚1.7884172028780231e+09 /loss/FRt2/assignment_loss‚5.757631078679511 /loss/FBt1/detection_loss‚2.1125689412815905 .loss/FBt3/detection_loss‚8.953810286063414 1loss/SRqqt2/assignment_loss‚2.788791595503341 2loss/SRqqt4/assignment_loss‚2.6065731694899408 /loss/FRt1/detection_loss‚2.9975673524956954 .loss/FBt2/detection_loss‚4.488537947340099 0loss/SRqqt1/detection_loss‚4.875707043842836 1loss/SRqqt3/detection_loss‚2.9687407051408012 /loss/FBt4/detection_loss‚2.5308798234099927 /loss/FRt3/assignment_loss‚5.374374580383301 0loss/FBt3/assignment_loss‚0.6998110225663717 .loss/FRt4/detection_loss‚2.016342833025815 0loss/SRqqt4/detection_loss‚1.254077482065298 2loss/SRqqt3/assignment_loss‚2.0816890197091324 0loss/FBt2/assignment_loss‚0.9332732068238935 .loss/FRt2/detection_loss‚2.983173168085991 _runtime‚ 702.615464722 _step‚75 0loss/FBt1/assignment_loss‚0.5101357056770385 1loss/SRqqt2/detection_loss‚3.2022102128627687 $loss/total_loss‚76.5147418852475 trainer/global_step‚949 /loss/FRt1/assignment_loss‚6.754814524399607 /loss/FRt4/assignment_loss‚4.523217857700505 1loss/FBt4/assignment_loss‚0.24937278865077725 epoch‚0 1loss/SRqqt1/assignment_loss‚2.738271388140592 /loss/FRt3/detection_loss‚3.1131701519091926[ßf—TïÎ 2loss/SRqqt3/assignment_loss‚2.0816890197091324 0loss/FBt2/assignment_loss‚0.9332732068238935 .loss/FRt2/detection_loss‚2.983173168085991 _runtime‚ 702.615464722 _step‚75 0loss/FBt1/assignment_loss‚0.5101357056770385 1loss/SRqqt2/detection_loss‚3.2022102128627687 $loss/total_loss‚76.5147418852475 trainer/global_step‚949 /loss/FRt1/assignment_loss‚6.754814524399607 /loss/FRt4/assignment_loss‚4.523217857700505 1loss/FBt4/assignment_loss‚0.24937278865077725 epoch‚0 1loss/SRqqt1/assignment_loss‚2.738271388140592 /loss/FRt3/detection_loss‚3.1131701519091926 % _timestamp‚1.7884172028780231e+09 /loss/FRt2/assignment_loss‚5.757631078679511 /loss/FBt1/detection_loss‚2.1125689412815905 .loss/FBt3/detection_loss‚8.953810286063414 1loss/SRqqt2/assignment_loss‚2.788791595503341 2loss/SRqqt4/assignment_loss‚2.6065731694899408 /loss/FRt1/detection_loss‚2.9975673524956954 .loss/FBt2/detection_loss‚4.488537947340099 0loss/SRqqt1/detection_loss‚4.875707043842836 1loss/SRqqt3/detection_loss‚2.9687407051408012 /loss/FBt4/detection_loss‚2.5308798234099927 /loss/FRt3/assignment_loss‚5.374374580383301 0loss/FBt3/assignment_loss‚0.6998110225663717 .loss/FRt4/detection_loss‚2.016342833025815 0loss/SRqqt4/detection_loss‚1.254077482065298K|Â#û’ð‚(:‡ µ©äÔ©ìª~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.002' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-^Ê:ñ‚(:¯ µ©äÔ©Ð disk.sda.out‚ 29.390625$ disk.nvme0n1.out‚2227.9072265625 disk.rbd0.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.33014678955078 network.recv‚333632 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.09375 disk.drbd2080.out‚0 disk.dm-1.out‚ 29.390625 disk.rbd1.out‚ 0.306640625 disk.drbd2080.in‚0 network.sent‚ 1.159108e+06 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.09375 disk.sda3.out‚ 29.390625! disk.dm-0.out‚2227.9072265625# memory_percent‚84.4625473022461 cpu‚25.05806851270401& disk.nvme0c0n1.out‚2227.9072265625& proc.memory.availableMB‚ 2545.65625) disk./.usagePercent‚16.92754073346118 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚252.2325897216797 disk.drbd1716.out‚0 disk.dm-0.in‚0.09375 proc.memory.rssMB‚ 2675.53125ªé§–’ò‚(:‡ Ä©äÔôô±~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.643' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1286 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BùrCó‚(:¸ Ä©äԛ𞀠disk.sda3.out‚ 29.9140625) disk./.usagePercent‚16.92833845293787$ disk.nvme0n1.out‚2268.1201171875 disk.drbd2265.in‚0 disk.sda.out‚ 29.9140625 disk.sda1.in‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 2545.66796875 disk.nvme0n1.in‚0.09375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.09375 disk.drbd2080.out‚0 disk.drbd2266.in‚0 cpu‚25.057667772432183 network.recv‚339943 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2268.1201171875 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚252.24447631835938 disk.dm-1.out‚ 29.9140625 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.306640625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0) proc.memory.percent‚16.33014678955078 network.sent‚ 1.209181e+06 disk.drbd2265.out‚0 disk.dm-0.in‚0.09375$ memory_percent‚84.46247577667236 proc.memory.rssMB‚ 2675.53125 disk.drbd2080.in‚0! disk.dm-0.out‚2268.1201171875 disk.sda1.out‚0 disk.dm-1.in‚0Õ5’ô‚(:‡ Ó©äÔÖߧ~ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Ÿ¢EHõ‚(:½ Ó©äÔ¶õ„€% disk.nvme0n1.out‚2311.38427734375 disk.rbd0.out‚0$ memory_percent‚84.46259498596191 proc.cpu.threads‚17$ disk./.usageGB‚252.2482147216797 disk.sda1.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375) proc.memory.percent‚16.33017063140869 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2675.53515625 disk.drbd1716.out‚0 disk.rbd1.out‚ 0.306640625 disk.sda.out‚ 30.5703125 network.sent‚ 1.214395e+06 network.recv‚343319* disk./.usagePercent‚16.928589340321423 disk.nvme0c0n1.in‚0.09375' disk.nvme0c0n1.out‚2311.38427734375 disk.sda3.in‚0 disk.sda3.out‚ 30.5703125 disk.drbd2265.in‚0" disk.dm-0.out‚2311.38427734375 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 2545.6484375 cpu‚25.057530506131823 disk.sda2.out‚0 disk.dm-1.out‚ 30.5703125 disk.sda.in‚0 disk.drbd1716.in‚0Æàœ]0ö+ _wandbruntime‚738  _runtime‚738ZãÊÅ÷¿ $ _timestamp‚1.788417238674301e+09 _runtime‚ 738.411957343 _step‚76 ' lr-AdamW/pg1‚0.00013938620361788899 ' lr-AdamW/pg2‚0.00013938620361788899 trainer/global_step‚999ScàÉøà trainer/global_step‚999 $ _timestamp‚1.788417238674301e+09 _runtime‚ 738.411957343 _step‚76 ' lr-AdamW/pg1‚0.00013938620361788899 ' lr-AdamW/pg2‚0.00013938620361788899L¨e0ù+ _wandbruntime‚738  _runtime‚7381Þ1·º'ú´O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4623949 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57190072504832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54919813948416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 738.412795524 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 819465575424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18534726 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884172386750386e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14023598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32539782 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872½ˆèVüû¸O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57189398906368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56369933330944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current]ùEüÂ$‚ 9326034944 _step‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32539782 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14023598 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55736940117394 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8173702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚61671369035264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32558324 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 170682368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56369933330944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 820121170944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61671392975360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54919830987776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 819465575424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57190072504832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚468 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 817109129618 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4623949 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57190072504832 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194622464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54919813948416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 738.412795524 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12797649 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8173700 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 819465575424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18534726 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57189398906368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚885 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884172386750386e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18516186 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4624417 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14023598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62533961392640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14023596 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚62533766770176 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 862568417280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 862397734912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 816454699410 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4628549 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57247244984832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18534640 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8181870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 739.091250494 _step‚78 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54974735118336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14037614 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14037614 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55791989213350 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57246571373056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32590820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8181870 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 739.091250494 _step‚78 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54974735118336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14037614 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14037614 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32590820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14037612 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12810417 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14037612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61733043713024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚62596459435520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 863415722496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 820285003264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 820940612096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18553206 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4629022 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 817925564790NÇ *i0ÿ+ _wandbruntime‚739  _runtime‚739‘ú.R€Ì /loss/FRt1/assignment_loss‚6.481271038884702 /loss/FRt3/assignment_loss‚6.123468586892793 .loss/FRt4/assignment_loss‚3.65007290653154 /loss/FRt1/detection_loss‚2.8575457852819692 .loss/FBt3/detection_loss‚5.896420478820801 1loss/SRqqt4/assignment_loss‚2.331536595334952 0loss/FBt3/assignment_loss‚0.6781126643663811 .loss/FRt4/detection_loss‚2.730126171719794 1loss/SRqqt4/detection_loss‚1.0568273484967141 epoch‚0 trainer/global_step‚999 /loss/FRt2/assignment_loss‚5.300250479153225 /loss/FBt2/detection_loss‚3.7518981218338014 %loss/total_loss‚73.21896218048472 2loss/SRqqt2/assignment_loss‚2.5926831354697546 1loss/SRqqt3/detection_loss‚3.0583421804010866 .loss/FBt1/detection_loss‚4.075650244022494 _runtime‚ 739.09235572 _step‚79 2loss/SRqqt1/assignment_loss‚2.7766191164652505 0loss/FBt2/assignment_loss‚0.9170376181602478 1loss/FBt4/assignment_loss‚0.28080718440560304 /loss/FRt2/detection_loss‚2.4286617381232127 0loss/SRqqt1/detection_loss‚5.529673266741964 /loss/FBt4/detection_loss‚2.0055237312002907 0loss/FBt1/assignment_loss‚0.5810860375252863 /loss/FRt3/detection_loss‚3.8550801864176085 % _timestamp‚1.7884172393549278e+09 2loss/SRqqt3/assignment_loss‚2.1532040334772318 1loss/SRqqt2/detection_loss‚2.1070635307580234êSÃVÐ /loss/FRt1/assignment_loss‚6.481271038884702 /loss/FRt3/assignment_loss‚6.123468586892793 .loss/FRt4/assignment_loss‚3.65007290653154 /loss/FRt1/detection_loss‚2.8575457852819692 .loss/FBt3/detection_loss‚5.896420478820801 1loss/SRqqt4/assignment_loss‚2.331536595334952 0loss/FBt3/assignment_loss‚0.6781126643663811 .loss/FRt4/detection_loss‚2.730126171719794 1loss/SRqqt4/detection_loss‚1.0568273484967141 epoch‚0 trainer/global_step‚999 /loss/FRt2/assignment_loss‚5.300250479153225 /loss/FBt2/detection_loss‚3.7518981218338014 %loss/total_loss‚73.21896218048472 2loss/SRqqt2/assignment_loss‚2.5926831354697546 1loss/SRqqt3/detection_loss‚3.0583421804010866 .loss/FBt1/detection_loss‚4.075650244022494 _runtime‚ 739.09235572 _step‚79 2loss/SRqqt1/assignment_loss‚2.7766191164652505 0loss/FBt2/assignment_loss‚0.9170376181602478 1loss/FBt4/assignment_loss‚0.28080718440560304 /loss/FRt2/detection_loss‚2.4286617381232127 0loss/SRqqt1/detection_loss‚5.529673266741964 /loss/FBt4/detection_loss‚2.0055237312002907 0loss/FBt1/assignment_loss‚0.5810860375252863 /loss/FRt3/detection_loss‚3.8550801864176085 % _timestamp‚1.7884172393549278e+09 2loss/SRqqt3/assignment_loss‚2.1532040334772318 1loss/SRqqt2/detection_loss‚2.1070635307580234O}wg’‚‚(:‡ â©äÔð•£~ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µf]ÿÛƒ‚(:¸ â©äÔ»ó«€ disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.drbd2265.out‚0" disk.dm-0.out‚2362.32470703125 disk.dm-1.out‚ 31.046875 disk.rœÂæhbd1.in‚0 disk.sda.out‚ 31.046875 disk.sda1.out‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚2362.32470703125 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0% disk.nvme0n1.out‚2362.32470703125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚84.46292877197266& proc.memory.availableMB‚ 2545.59375# proc.memory.rssMB‚ 2675.53515625 network.sent‚ 1.220342e+06 disk.drbd2080.in‚0 disk.rbd1.out‚ 0.306640625) proc.memory.percent‚16.33017063140869 cpu‚25.05736464735576 disk.drbd2266.in‚0 disk.sda3.out‚ 31.046875 proc.cpu.threads‚17* disk./.usagePercent‚16.929385011737843 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375 disk.rbd0.out‚0 network.recv‚347625% disk./.usageGB‚252.26007080078125ÿ‚\H„‚(:„ ñ©äÔó“¤~ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 !¢?…‚(:´ ñ©äÔâ·ˆ disk.dm-1.out‚31.8125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚2405.64501953125 disk.sda3.out‚31.8125( proc.memory.availableMB‚ 2545.6953125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2675.53515625 disk.rbd1.in‚0 disk.sda.out‚31.8125 disk.sda2.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.33017063140869 cpu‚25.05717798274962 network.sent‚ 1.269873e+06 disk.dm-1.in‚0% disk.nvme0n1.out‚2405.64501953125 network.recv‚352997 disk.drbd2080.out‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.306640625 disk.sda3.in‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚2405.64501953125 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.nvme0c0n1.in‚0.09375* disk./.usagePercent‚16.929297713168665 disk.drbd2080.in‚0 disk.sda.in‚0$ memory_percent‚84.46230888366699% disk./.usageGB‚252.25876998901367 disk.sda1.in‚0á润0†+ _wandbruntime‚775  _runtime‚775þ¨Å‡¿ ' lr-AdamW/pg2‚0.00014636249008525076 trainer/global_step‚1049 $ _timestamp‚1.788417275512495e+09 _runtime‚ 775.25002916 _step‚80 ' lr-AdamW/pg1‚0.00014636249008525076|X·ãɈà ' lr-AdamW/pg1‚0.00014636249008525076 ' lr-AdamW/pg2‚0.00014636249008525076 trainer/global_step‚1049 $ _timestamp‚1.788417275512495e+09 _runtime‚ 775.25002916 _step‚80P1‡‚ž0‰+ _wandbruntime‚775  _runtime‚775ÿö蟻'ŠµO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58522981984802 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 775.250925161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 905627977216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 857944469026 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 857290038818 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8582100 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65659533895680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64753929858560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59187585273344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14724396 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59187603276288 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460778 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 860451381248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚81 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 860451381248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚81 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4855754 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 861106976768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60048036654592 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58522981984802 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57665037515776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 905792368128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60048036654592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8582102 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59187603276288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 775.250925161 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14724398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30Q§n‰0Œ+ _wandbruntime‚775  _runtime‚7758«>å$¿M J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59243956315136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚82 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 861926417920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 861926417920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57719924606976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 775.956689078 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59243938312192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64815556656128 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048P¯W/à704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8590268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692£‘ØðÉ&ŽÃM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884172762187579e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1049 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60105209121280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19479258 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 861926417920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 861926417920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13450734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57719924606976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14738412 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64815580596224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14738412 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 906495420416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚ 775.956689078 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57719941646336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19479258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59243938312192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 861270809088 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚58578031080758 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64815556656128 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 858106473782 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58578702550534 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34217672 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34199104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34199104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4860918 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8590268 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19460692 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8590270 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60105882733056 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59243938312192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59243956315136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34217672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 162280448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚82 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14738414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59243956315136 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14738414 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 858760904198 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948868385280 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 # _timestamp‚1.78841731229689e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 trainer/global_step‚1099 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20386830 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5087676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 898125378226 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚85 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 949032776192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5088153 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 812.034600073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20386830 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68785499517952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62906674402816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚62907348001280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8990500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 188331008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚62906674402816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15425198 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚479 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15425196 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 902092782592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15425198 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14078176 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 948868385280 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61309023852210 # _timestamp‚1.78841731229689e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚68785311186944 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 898779808434 trainer/global_step‚1099 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62005237215744 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freœÝFLïed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20386830 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15425196 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5087676 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60410244043776 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 898125378226 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚85 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62005255218688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 949032776192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5088153 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 812.034600073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20386830 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚68785499517952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67836442801664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 901437187072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35812028 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚477 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8990502 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60410227004416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62005255218688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14078655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477UöôÓ;0ž+ _wandbruntime‚812  _runtime‚812o|Å&Ÿ¿M WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68847831384064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5092809 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20405310 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62964520481280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35825956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61364072948166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60465148174336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62061590254592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15439212 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 902256614912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 812.727702599 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8998670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62963846869504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62061590254592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20405310 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35844524 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884173129896386e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 902256614912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚1099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15439212 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872:°†´É& ÃM O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚62964520481280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35825956 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62964520481280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67898117479424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61364072948166 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60465148174336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62061590254592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15439212 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 902256614912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 812.727702599 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8998670 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20386744 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚62963846869504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14091479 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68848013410304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62061608257536 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62061590254592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 949895930880 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20405310 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15439214 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 158086144 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 902912223744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 898941813190 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35844524 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62061608257536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884173129896386e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 902256614912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚1099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15439212 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61364744417942 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14091021 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 949737844736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67898093539328 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68847831384064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35825956 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5092809 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20405310 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35844524 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 182026240Vn‰`0¡+ _wandbruntime‚812  _runtime‚812oN90O¢É 2loss/SRqqt1/assignment_loss‚3.0921603991435123 /loss/FBt4/detection_loss‚2.3171960210792073 /loss/FBt1/assignment_loss‚0.487118444511933 1loss/FBt4/assignment_loss‚0.19527698354599837 0loss/SRqqt4/detection_loss‚1.258942646022542 %loss/total_loss‚64.72762121524053 0loss/FBt3/assignment_loss‚0.5612428456348425 /loss/FRt2/detection_loss‚2.7112479514545864 0loss/SRqqt1/detection_loss‚3.437178541834538 /loss/FBt2/detection_loss‚0.2692355388832807 0loss/FRt1/assignment_loss‚6.0492453336715695 .loss/FRt4/assignment_loss‚3.73434601579943 1loss/SRqqt3/assignment_loss‚2.257614258909598 0loss/FBt2/assignment_loss‚0.8042459433728998 1loss/SRqqt2/detection_loss‚2.2829048746037035 _runtime‚ 812.72869404 /loss/FRt3/detection_loss‚2.9874879681482547 $ _timestamp‚1.788417312991252e+09 /loss/FRt3/assignment_loss‚5.876524925231934 _step‚87 trainer/global_step‚1099 /loss/FRt2/assignment_loss‚4.636217249764337 1loss/SRqqt4/assignment_loss‚2.125265554889389 .loss/FBt1/detection_loss‚3.422016067725573 .loss/FBt3/detection_loss‚5.391184075673421 0loss/SRqqt3/detection_loss‚3.350928332656622 epoch‚0 /loss/FRt1/detection_loss‚2.9890692114830015 2loss/SRqqt2/assignment_loss‚2.3110740229768574 .loss/FRt4/detection_loss‚2.179898008223503„®†S£Í 0loss/SRqqt3/detection_loss‚3.350928332656622 epoch‚0 /loss/FRt1/detection_loss‚2.9890692114830015 2loss/SRqqt2/assignment_loss‚2.3110740229768574 .loss/FRt4/detection_loss‚2.179898008223503 2loss/SRqqt1/assignment_loss‚3.0921603991435123 /loss/FBt4/detection_loss‚2.3171960210792073 /loss/FBt1/assignment_loss‚0.487118444511933 1loss/FBt4/assignment_loss‚0.19527698354599837 0loss/SRqqt4/detection_loss‚1.258942646022542 %loss/total_loss‚64.72762121524053 0loss/FBt3/assignment_loss‚0.5612428456348425 /loss/FRt2/detection_loss‚2.7112479514545864 0loss/SRqqt1/detection_loss‚3.437178541834538 /loss/FBt2/detection_loss‚0.2692355388832807 0loss/FRt1/assignment_loss‚6.0492453336715695 .loss/FRt4/assignment_loss‚3.73434601579943 1loss/SRqqt3/assignment_loss‚2.257614258909598 0loss/FBt2/assignment_loss‚0.8042459433728998 1loss/SRqqt2/detection_loss‚2.2829048746037035 _runtime‚ 812.72869404 /loss/FRt3/detection_loss‚2.9874879681482547 $ _timestamp‚1.788417312991252e+09 /loss/FRt3/assignment_loss‚5.876524925231934 _step‚87 trainer/global_step‚1099 /loss/FRt2/assignment_loss‚4.636217249764337 1loss/SRqqt4/assignment_loss‚2.125265554889389 .loss/FBt1/detection_loss‚3.422016067725573 .loss/FBt3/detection_loss‚5.391184075673421WÕ¹„’¤‚(:‡ ­ªäÔ»¯Œ~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.794' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äTkE¥‚(:º ­ªäÔÈ disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 34.6484375) proc.memory.availableMB‚ 2545.57421875 network.sent‚ 1.336968e+06 disk.rbd0.in‚0 disk.drbd2080.in‚0) disk./.usagePercent‚16.93374686810623 disk.nvme0c0n1.in‚0.09375 disk.drbd2080.out‚0 disk.dm-1.out‚ 34.6484375 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.percent‚16.3301944732666 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375" disk.dm-0.out‚2577.36083984375 disk.sda1.in‚0% disk.nvme0n1.out‚2577.36083984375 cpu‚25.05638158022039 disk.sda3.out‚ 34.6484375 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚2577.36083984375 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2675.5390625 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.09375% disk./.usageGB‚252.32506561279297 disk.sda2.out‚0 disk.rbd1.out‚ 0.455078125# memory_percent‚84.4630479812622 network.recv‚371571ITŸý¦‚(:’ ¼ªäÔÔ ‘~ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.945' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.788999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 2G§‚(:¼ ¼ªäÔú؇€ disk.nvme0c0n1.in‚0.09375 disk.drbd2266.out‚0 proc.cpu.threads‚17% disk./.usageGB‚252.33235931396484 disk.sda.out‚ 34.99609375' disk.nvme0c0n1.out‚2635.37744140625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.sent‚ 1.387142e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0" disk.dm-0.out‚2635.37744140625 disk.sda1.in‚0 disk.sda3.out‚ 34.99609375 disk.drbd2266.in‚0 cpu‚25.05603906335639 disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0% proc.memory.availableMB‚ 2545.5625 disk.drbd1716.out‚0% disk.nvme0n1.out‚2635.37744140625 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 34.99609375 disk.rbd1.out‚ 0.4873046875( proc.memory.percent‚16.3301944732666 network.recv‚377836 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚84.46311950683594" proc.memory.rssMB‚ 2675.5390625* disk./.usagePercent‚16.934236354511697æäç?0¨+ _wandbruntime‚848  _runtime‚848åAVÅ©¿ trainer/global_step‚1149 % _timestamp‚1.7884173491237335e+09 _runtime‚ 848.861259962 _step‚88 & lr-AdamW/pg1‚0.0001603150630199744 & lr-AdamW/pg2‚0.0001603150630199744à†·Lɪà & lr-AdamW/pg1‚0.0001603150630199744 & lr-AdamW/pg2‚0.0001603150630199744 trainer/global_step‚1149 % _timestamp‚1.7884173491237335e+09 _runtime‚ 848.861259962 _step‚88XIùl„0«+ _wandbruntime‚848  _runtime‚848eŽÄEh¬µO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992065942016 XIDeviceStat>UìWS%sMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9398902 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 943078588416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9398900 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37438880 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21312882 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚1149 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71911045626880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚89 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 939615147842 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65765985749504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 942422992896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884173491243994e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64094394250050 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16125998 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21294342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63155433532416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16125996 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70919003624960ã‹l!¿'­¹O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16125998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63155433532416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16125996 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65765985749504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64822907161088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 938960717634 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65765312151040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70919003624960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64822889158144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14719787 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 992065942016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9398902 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 175748096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643691477 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654430208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 943078588416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚64822907161088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63155450571776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 942422992896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9398900 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 655595520 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37438880 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21312882 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚1149 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71911045626880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70918979684864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚89 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 655595520 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71911221374976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 939615147842 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37420338 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 992217750016 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21312882 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65765985749504 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 942422992896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884173491243994e+09 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64094394250050 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16125998 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚64822889158144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21294342 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5320885 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16125996 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5320423 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64095065719618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21294342 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568Y7ie30®+ _wandbruntime‚849  _runtime‚849y¨éVÅ&¯¿M K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚ 849.554972233 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65823158229504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 trainer/global_step‚1149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 939777152598 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65822484617728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚90 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889—ýÞ¢Ö °ÃM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16140012 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚64879260199936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21331362 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14732558 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 943898029568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861845973 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37452808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9954819072 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 943242420736 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5325019 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64879242196992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71973740571648 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65823158229504 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993086209024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 185637376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63210337662976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/acA÷ƒótive_bytes.small_pool.current‚ 655608832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚469 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9407070 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 939777152598 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65822484617728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚64879242196992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 943898029568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37471376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 190414848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 943242420736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚471 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 166474752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 992919734272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚90 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37471376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21312796 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 673611776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 940431583014 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21331362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14732087 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚ 849.554972233 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21312796 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9407068 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚64879260199936 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70980630422528 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63210354702336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65823158229504 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16140014 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71973550156800 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 trainer/global_step‚1149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37452808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚1199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74001540508160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 980450487250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15359319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 _runtime‚ 885.433276515 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚93 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67640541100544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22220394 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983408798720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15359799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630¨2,P†¿»O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16826798 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67640559103488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16826798 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66881107587026 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66880436117458 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884173856954968e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75036850610688 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 673598464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 185637376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65900640060416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65900657099776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863048704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39047190 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9954819072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22238934 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 984064394240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 trainer/global_step‚1199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 983408798720 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9807302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74001540508160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚478 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚75036662279680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 980450487250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68623949899264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863048704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16826796 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22220394 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22238934 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15359319 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861845973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 164390912 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 979796057042 _runtime‚ 885.433276515 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16826796 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚93 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67640541100544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22220394 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983408798720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9954819072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15359799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671469568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68624623497728 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68624623497728 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68623949899264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1035145711616 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 984064394240 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9807300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 673598464 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74001516568064]D V0À+ _wandbruntime‚886  _runtime‚886ÿÝO¨Æ&ÁÀM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚1199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74063191245824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68681795977728 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 984883835392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68681795977728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1036172769792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 _runtime‚ 886.128140261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39079660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9815468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16840812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5556628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980612492006 _step‚94 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984228226560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414VÄWÊ&ÂÄM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9815468 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984228226560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16840812 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15372574 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654430416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 655608832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16840812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67696912142336 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5556628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 984883835392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980612492006 _step‚94 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9954819072 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39098228 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16840814 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74063167305728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 984228226560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863048704 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68681122365952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671469776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65955544190976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67696912142336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22238848 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 673611776 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22257414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1036010489344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39079660 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚1199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74063191245824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66936156682982 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39098228 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 186220544 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9815470 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15372096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68681795977728 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 984883835392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68681795977728 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863048704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75099364015616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1036172769792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66936828152758 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643691477 _runtime‚ 886.128140261 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚65955561230336 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22257414 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22238848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39079660 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944^ëݳ0Ã+ _wandbruntime‚886  _runtime‚886Ò-bÏRÄÌ %loss/total_loss‚65.86673699630862 1loss/SRqqt4/detection_loss‚1.1114008784615657 /loss/FRt1/assignment_loss‚6.394368080865769 1loss/SRqqt1/assignment_loss‚3.068450555205345 2loss/SRqqt4/assignment_loss‚2.0332218628554117 /loss/FRt1/detection_loss‚2.4856904234204973 2loss/SRqqt2/assignment_loss‚2.2328462527722728 0loss/SRqqt2/detection_loss‚2.409997819029555 0loss/FRt2/assignment_loss‚4.8357098241647085 2loss/SRqqt3/assignment_loss‚2.5239520420630774 .loss/FRt3/detection_loss‚3.540066581347893 .loss/FRt4/detection_loss‚2.029029206587718 0loss/SRqqt1/detection_loss‚3.428727626800537 .loss/FBt1/detection_loss‚5.173664267341208 /loss/FBt3/detection_loss‚4.0552639842033384 .loss/FBt4/detection_loss‚1.533909389632754 0loss/FBt2/assignment_loss‚0.9321353062987328 0loss/FBt4/assignment_loss‚0.2715142626327349 /loss/FRt2/detection_loss‚2.3037704050540926 epoch‚0 % _timestamp‚1.7884173863917232e+09 _step‚95 /loss/FRt4/assignment_loss‚3.514740216732025 0loss/FBt3/assignment_loss‚0.6480679973909295 0loss/SRqqt3/detection_loss‚2.948748761054241 _runtime‚ 886.129160075 /loss/FRt3/assignment_loss‚6.267944039969609 0loss/FBt1/assignment_loss‚0.6111739249566475 /loss/FBt2/detection_loss‚1.5123432874679565 trainer/global_step‚11995E KVÅÐ %loss/total_loss‚65.86673699630862 1loss/SRqqt4/detection_loss‚1.1114008784615657 /loss/FRt1/assignment_loss‚6.394368080865769 1loss/SRqqt1/assignment_loss‚3.068450555205345 2loss/SRqqt4/assignment_loss‚2.0332218628554117 /loss/FRt1/detection_loss‚2.4856904234204973 2loss/SRqqt2/assignment_loss‚2.2328462527722728 0loss/SRqqt2/detection_loss‚2.409997819029555 0loss/FRt2/assignment_loss‚4.8357098241647085 2loss/SRqqt3/assignment_loss‚2.5239520420630774 .loss/FRt3/detection_loss‚3.540066581347893 .loss/FRt4/detection_loss‚2.029029206587718 0loss/SRqqt1/detection_loss‚3.428727626800537 .loss/FBt1/detection_loss‚5.173664267341208 /loss/FBt3/detection_loss‚4.0552639842033384 .loss/FBt4/detection_loss‚1.533909389632754 0loss/FBt2/assignment_loss‚0.9321353062987328 0loss/FBt4/assignment_loss‚0.2715142626327349 /loss/FRt2/detection_loss‚2.3037704050540926 epoch‚0 % _timestamp‚1.7884173863917232e+09 _step‚95 /loss/FRt4/assignment_loss‚3.514740216732025 0loss/FBt3/assignment_loss‚0.6480679973909295 0loss/SRqqt3/detection_loss‚2.948748761054241 _runtime‚ 886.129160075 /loss/FRt3/assignment_loss‚6.267944039969609 0loss/FBt1/assignment_loss‚0.6111739249566475 /loss/FBt2/detection_loss‚1.5123432874679565 trainer/global_step‚1199_è kä’Æ‚(:‡ øªäÔúlj~ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.307' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.2614 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã!XmEÇ‚(:º øªäÔüÉ«€! disk.dm-0.out‚2803.4794921875 disk.dm-1.out‚ 37.33203125 disk.sda.out‚ 37.33203125 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 37.33203125 network.sent‚ 1.454206e+06 disk.dm-0.in‚0.09375 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 disk.dm-1.in‚0& disk.nvme0c0n1.out‚2803.4794921875 disk.drbd2265.out‚0$ memory_percent‚84.97452735900879 network.recv‚396309 disk.drbd2080.in‚0 disk.sda.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.sda3.in‚0 proc.memory.rssMB‚ 2758.78125 disk.drbd2080.out‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚2803.4794921875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 2461.7734375* disk./.usagePercent‚16.935790832259602% disk./.usageGB‚252.35552215576172 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.09375) proc.memory.percent‚16.83826446533203 cpu‚25.092073684372554½+†ð‘È‚(:† ‡«äÔÖûŠ~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'›NŽ@É‚(:µ ‡«äÔÖÝ‚€ disk.dm-1.out‚37.9375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚0.09375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚2844.97998046875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.out‚0 network.recv‚400998 disk.sda.in‚0$ memory_percent‚85.08694171905518 network.sent‚ 1.503192e+06 disk.nvme0c0n1.in‚0.09375 disk.drbd2265.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2443.35546875% disk./.usageGB‚252.36606979370117' disk.nvme0c0n1.out‚2844.97998046875 disk.sda2.in‚0 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.sda.out‚37.9375# proc.memory.rssMB‚ 2777.38671875* proc.memory.percent‚16.951823234558105* disk./.usagePercent‚16.936498693091778 disk.sda3.out‚37.9375 disk.drbd2080.out‚0" disk.dm-0.out‚2844.97998046875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 cpu‚25.17084612865825ø@S|‘Ê‚(:† –«äÔéóÈ~ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚179.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ý%¡?Ë‚(:´ –«äÔ‰‡³€ disk.dm-0.in‚0.09375 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.09375 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0$ memory_percent‚84.95938777923584) proc.memory.availableMB‚ 2464.25390625 network.sent‚ 1.509268e+06 disk.nvme0n1.in‚0.09375 disk.dm-0.out‚ 2891.96484375) proc.memory.percent‚16.84091091156006 proc.cpu.threads‚17# proc.memory.rssMB‚ 2759.21484375% disk./.usageGB‚252.36498260498047$ disk.nvme0c0n1.out‚ 2891.96484375 disk.sda2.out‚0 disk.sda3.out‚ 38.4765625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 network.recv‚405595 disk.sda2.in‚0 disk.dm-1.out‚ 38.4765625" disk.nvme0n1.out‚ 2891.96484375 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda.out‚ 38.4765625 cpu‚25.24632391623386* disk./.usagePercent‚16.936425730944517dÍÉtÌbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPi·Î~Íby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP¨àÿ¦yÎbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP*ßÂtÏbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP)…NoÐbj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JP9_£ªsÑbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPû±§×jÒbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPGëØKjÓbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPâ‚TœvÔbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPçÆhÕbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP€ŸýqÖbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JP—Êzåp×bk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPit¹$yØbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPpqHtÙbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÔûgnÚbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPPþŒfkÛbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP¤=ÄuÜbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPù’Qk]ÝbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JPÓqû@mÞbh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JP2²Ó‹rßbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPm°žYràbm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPCMyábt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPD– ®iâbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPø5߀qãbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPl oäbj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JP^çã¦nåbi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP),?0æbl IDeviceStatsMonitor.on_validation_batch_st˜np6Aart/reserved_bytes.all.allocated"trainer/global_step2:JPfà ÿwçbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP9¤ÉAlèbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JPÜSzÏeéb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPžk mêbh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JP’·¸Ùyëbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPŠ…k¹wìbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPøäj|íbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP2ûyBoîbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP%‘Ÿ¨xïbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP[A„.uðbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP'rñbm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPHžœevòbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPoŠ:vóbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP¥¹•|ôbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP'@Ñsõbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPeÕQvmöbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPS&ìÁy÷bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP…ï¥âwøbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP¥®oëvùbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP¹ùÁqúbl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPÄ0fûba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPÉ kübf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JPÀ+‚evýbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP, ¿žtþbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP·€+DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP Ñ:®i™bd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPëÙršbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JP0}zp›bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPV{!~œby VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPA6±ýzbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP.‘yìwžbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPN55nŸbi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JPíP)\k bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JPEЦén¡bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP«þ(t¢bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP‰…¥h£bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPÑôÛm¤bh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPUôún¥bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JPÔžl¦bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPcNƒot§bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPšå¦4e¨b` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPMæNMg©bb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP1Itªbo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPZ¶Œt«bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPö=>ól¬bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JPR¬«ìx­bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPyýŠ*e®b` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPj»·m¯bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPXçÏíl°bg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPö^ýt±bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPƒªàm²bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JP"9§Õd³b_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚22781251 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863048704 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22766060 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚77535980766208 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507950476 % _timestamp‚1.7884174408749003e+09 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚76473934886400 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1208 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17496450 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062021939712 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1061910387712 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚69185032382330 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22630 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863048704 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚ 940.61299288 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22766060 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492427776 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17496452 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚22781251 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863048704 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚77535845274112 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚22766060 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚77535980766208 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1010164851200 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚17496450 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507950476 % _timestamp‚1.7884174408749003e+09 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚16039249 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚10207678 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚76473934886400 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚156 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚1208 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1005956801402 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40277703 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚17496450 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062021939712 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1061910387712 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚69185032382330 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚22630 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71018415323648 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863048704 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚ 940.61299288 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚22766060 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492427776 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚17496452 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚5831573 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚40277703 BØEpÑbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP!:¡@rÒbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÒ|XqÓbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP JáuÔbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÉ$[rÕbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP—Ø  rÖbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP÷ýÜq×bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP/ƒ“3pØbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP°ÉÄvÙbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP\à<vÚbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP ˆˆqÛbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPÿ9_ókÜbf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JPî”k"cÝb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPxïà nÞbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPI.nßbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JPNßì‡jàbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPõaèxábs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP]rA«qâbl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPrà¯ùtãbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JPdÊ$okäbf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP¥kiåbd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JP›u+;wæbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPF¶>9rçbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPy“çsèbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP´–ˆKsébn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPñ&\lêbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPEn÷›uëbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPÐ’­âgìbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JP«„>míbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JP²¿3jîbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JP ÉÙ’bïb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP¦˜üolðbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP<d|hñbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JPÇ4«kòbf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JPhíÈcób^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JP'û…kôbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPu!¢põbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPá[épöbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPp!‡n÷bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JP‚¨I#døb_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP2åÌl¡bg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPnFÚg¢bb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP5gi£bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JP¡¤‚.r¤bm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPþÍs¥bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP²¸Åªq¦bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP!_Î[e§b` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPîøKk¨bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPŸâ3t©bo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP#LÁäuªbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPn÷õ¶m«bh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP¶5„¦n¬bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPåÈ0ýz­bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPêŒkKz®bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÙ“Ì•p¯bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP©â¥ i°bd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP>%ŒÆu±bp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP Ò¯¤d²b_ DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP›ž RrÁbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPH1[ó|Âbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP­0xšeÃb` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP!f®œwÄbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPtžèÕ0Å+ _wandbruntime‚941  _runtime‚941x½dQ6)Æ°R JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9954819072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚16050559 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507950476 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 % _timestamp‚1.7884174418077445e+09 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40295811 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚68214369340416 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9954819072 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71056120909824 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚69221197701702 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚77574221626368 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062387066368 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1006811321926 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚70045590931968 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚70045572929024 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚71056120909824 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚10214182 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚77574086134272 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510430720 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 941.545656241 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135492096 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1006320410810–ÿŽFÇ´R VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1010529977856 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚17507256 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚22773362 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚157 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚69220689751226 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1208 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚70045590931968 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1010529977856 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1010037550080 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚22788553 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚76511834560000 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚5836379 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚22630 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 % _timestamp‚1.7884174418077445e+09 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71055610479104 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40295811 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492427776 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚68214369340416 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9954819072 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚71056120909824 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚17507258 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111552000 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚69221197701702 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15193 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚77574221626368 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚10214180 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1062387066368 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 1006811321926 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚70045590931968 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚68214386379776 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚70045572929024 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863048704 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861845973 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peTÖ:gô ak‚ 9326034944 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚71056120909824 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚10214182 ZNDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚5836534 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚77574086134272 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643691477 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚40280618 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚22788553 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510430720 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚76511810619904 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71055610479104 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 941.545656241 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 135492096 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1006320410810 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492427776 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚40280618 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9954819072 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚16050559 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1010037550080 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507950476 B;èb6 jet/accuracy_1_of_3"trainer/global_step2:JPÝ Û#;éb6 jet/accuracy_5_of_6"trainer/global_step2:JPëßkJêbE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPœ¡þHëbC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPÀWqóNìbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPˆ·ŽMNíbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP%ÛÆÞNîbI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPËAd5;ïb6 jet/accuracy_1_of_2"trainer/global_step2:JPE‹Ê=HðbC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP½Ú£JñbE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP{WhSHòbC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP“ ¶NóbI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¤ç®ÄJôbE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPW¨=ÌJõbE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPfãßmJöbE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP žÒ´J÷bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÁŒAßHøbC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP»_•HùbC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÕüTJúbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP:é]»HûbC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP9;õðNübI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP<ò(!JýbE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPJŒ«NþbI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÚžJÿbE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŒAGN€bI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPˆœ·BJbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPaS8—J‚bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP 58ƒb3 particle/f_score"trainer/global_step2:JPÇýØJ„bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP5C9CJ…bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPƒ—`ØJ†bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâ屟J‡bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPßpáJˆbE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPéR ÞJ‰bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP>ý1ÙNŠbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPêïc7J‹bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP)µBH² bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP­¶¬;³ b6 jet/accuracy_5_of_5"trainer/global_step2:JPGøfÛJ´ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP9Jœî;µ b6 jet/accuracy_1_of_6"trainer/global_step2:JPÀy ž;¶ b6 jet/accuracy_4_of_6"trainer/global_step2:JP^_H· bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPeŒ¸ÑN¸ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP§ +J¹ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÒðû<º b7 particle/specificity"trainer/global_step2:JP·„ÄXJ» bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP)åÃîH¼ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPD‰8J½ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP&ÎåH¾ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP|åÔQH¿ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP»¿‹‚NÀ bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPôòHÁ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¡XÞ.H bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPqmvò;à b6 jet/accuracy_4_of_4"trainer/global_step2:JPS€ÅôJÄ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPDÍHÅ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP$b2JÆ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP–±U¹HÇ bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPG!ÑJJÈ bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPËÄ:nJÉ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPΣJÊ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÿ‰¨?NË bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP„ã£$;Ì b6 jet/accuracy_1_of_7"trainer/global_step2:JPòÄ1;Í b6 jet/accuracy_5_of_7"trainer/global_step2:JPÙ¨§µHÎ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¤ž¿JÏ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPƒ«HÐ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP]$ HÑ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPAØ NÒ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP–ðu™JÓ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP÷ÙºNÔ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP˜Ù¾JÕ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPÊ&œcHÖ bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¾Ê 8N× bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP`kM—NØ bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP"O©U@Ù b; particle/accuracy_3_of_7"trainer/global_step2:JPÀîߤJÚ bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPUÝÍ\HÛ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÁ£NÜ bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÁq¡OJÝ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP à;JÞ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÚÌ'ðJß bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÿßð†Hà bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP|°AJá bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP(6§Jâ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPöicúJã bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÏG]%Jä bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÀVNå bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP¤wuJæ bE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPçú"Hç bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPîð¬iJè bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPz 55Jé bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP˜š~ãJê bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP§10Jë bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP–rêôNì bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP…×üÿJí bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPqŽ/^Jî bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÒy¥(Hï bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP›DÛ÷Jð bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP˜ßá‹Nñ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPâIlžNò bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPö(ÕJó bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPd†ŽJô bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP»í›Hõ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÇÓ€eNö bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÅP"J÷ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPSfÚHø bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¥y…—Nù bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPA[þ%Nú bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPðÉŠHû bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPi€‹Hü bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPë=à8Hý bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPßP Nþ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP•þŽNÿ bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPñ›ÚâN€ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¡I±ÇN bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPq¿Ú×J‚ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP -UHƒ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPåòJ„ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPRÚ©H… bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPvy¢CH† bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP8Ðm¦J‡ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPõ{XñNˆ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‡ŽÌN‰ bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP*r¾NŠ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP²—íûH‹ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÄ÷Ø!JŒ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP•+õÌN bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP$Ñ)¨NŽ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP=ªîŠN bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP@ë¨ÛN bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPJüóJ‘ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP¸ÐáéH’ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPâ¬J“ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP ÙN” bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP"÷¥J• bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP$M`¨J– bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPqÀ13H— bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÝÎS´J˜ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP×a8N™ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPvEt;š b6 jet/accuracy_1_of_4"trainer/global_step2:JP‘d­J› bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP¥–Jœ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP8ühÈH bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP¶%ËNž bI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP»6’NŸ bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP‚´;  b6 jet/accuracy_2_of_5"trainer/global_step2:JP +ŠJ¡ bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÀnI¢J¢ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPT;”¶H£ bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP½ÜP³H¤ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÖn3dN¥ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP<¬ÞN¦ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPþøÕN§ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP$¾æIN¨ bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPï©/H© bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPêÎWPNª bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¼p)}J« bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPýy;¬ b6 jet/accuracy_4_of_5"trainer/global_step2:JPno ËH­ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP‡ù„J® bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPéd÷H¯ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP J° bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP‰$aH± bC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPSòz†H² bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPé¹VNJ³ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP› cæN´ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPzÚ±Nµ bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP;»ÙæJ¶ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP9_©Ä@· b; particle/accuracy_4_of_5"trainer/global_step2:JPvêãJ¸ bE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP]QülH¹ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPò`A Nº bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPi¼ÅðH» bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP U’N¼ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP„#¶^N½ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¾°ËN¾ bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¼ÕÇþN¿ bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP žÅ=JÀ bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP~±K‚NÁ bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÍË„IJ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP€ÌßñNà bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPr§fNÄ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP‡)1NÅ bI &Purity/4FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPT? NÆ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPk¿åHÇ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP·ç-e@È b; particle/accuracy_2_of_6"trainer/global_step2:JPs¨¯ÞJÉ bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPç„?iJÊ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP–ÇœJË bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPlðH„ bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP”zcH… bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPxÞ J† bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP ¬Å‚J‡ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÖÆJˆ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPìùk”N‰ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPßš‡HŠ bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP C¥H‹ bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÚUIœNŒ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPî|¶¥@ b; particle/accuracy_2_of_2"trainer/global_step2:JP}TŘHŽ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP*h¥H bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP†õ‚N bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP´Éõ×@‘ b; particle/accuracy_1_of_4"trainer/global_step2:JPN—†€H’ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¨$If;“ b6 jet/accuracy_6_of_6"trainer/global_step2:JP^ꢩH” bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÉ s7J• bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPHè“mN– bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPIÀPN— bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP)ÒÈ«H˜ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP}²‰HH™ bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPàHÌ9Nš bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÕN– bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÌeamN— bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP1ÔéJ˜ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPá›—"J™ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPàë)¹Hš bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒ„Ç H› bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPKÍÆHœ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPJ¡ N bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPSøãJž bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPXï°ŒJŸ bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP.Ýi>;  b6 loss/val_total_loss"trainer/global_step2:JPOÖ„J¡ bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP©ðþN¢ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÁ¹õJ£ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPc·¶H¤ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÆÖ&H¥ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP#)ˆ@H¦ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP*^¸{N§ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÂ3–N¨ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÙÊ„ÖN© bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPPo~žNª bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPa ûkH« bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP,uªSH¬ bC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP>uXH­ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¾àH® bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPæp‚H¯ bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPŒ¿ÁN° bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPþ äcH± bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔ>ÓN² bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPME5H³ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JPÙ¨aH´ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPÌ/õuJµ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPd téH¶ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPNÕ H· bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPxÉ€;N¸ bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÎœ,¾N¹ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPž"ä¦Jº bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPl®û9H» bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPúY«J¼ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP˜*=ˆN½ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP3ÕJ¾ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPêÆW£H¿ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP“êL­HÀ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP±2—`HÁ bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPáyh…J bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÄ|H±Nà bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPA&]CNÄ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP2…õ[HÅ bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPæ‹L’JÆ bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPLAÎÞJÇ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPüü3ÌJÈ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPÿƒiHÉ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP,$¨ùHÊ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜI,ÆNË bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPè§ÊžJÌ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPŽ¹o(NÍ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPâ!;Î b6 jet/accuracy_1_of_1"trainer/global_step2:JPU±îj@Ï b; particle/accuracy_1_of_3"trainer/global_step2:JPòlÙÏJÐ bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPÇM'¸HÑ bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP0Æ{NNÒ bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPgß=ÜNÓ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPš`HÔ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPª.Ò¨NÕ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP ]JÖ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPçhe‚G× bB validation_average_jet_accuracy"trainer/global_step2:JPQa6NØ bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPá\¤ JÙ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPtÖžJÚ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPWµJÛ bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP°»Ç_NÜ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP†xí‰@Ý b; particle/accuracy_1_of_1"trainer/global_step2:JP-÷äHÞ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPq&.Hß bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¸§®Nà bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÔaÔ™Há bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP<·Jâ bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPödnÀNã bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÏóµ6Jä bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPˆù–Nå bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÝP‚è@æ b; particle/accuracy_2_of_4"trainer/global_step2:JPÛ>—hHç bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP´ð–Jè bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPäŸÁNé bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP{€6¼Nê bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPµ­®Në bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP?ùñ;ì b6 jet/accuracy_2_of_3"trainer/global_step2:JP8±‚Jí bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPW…ÿúJî bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPJU#¹Jï bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPBXo Nð bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPâNª:Jñ bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPWµç¢Jò bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPÜ=[ûJó bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPº½/DJô bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP[±ÌJõ bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÙ,ANJö bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPYvX?J÷ bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP½WáJNø bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPc ØÜJù bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP˜•ZoHú bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPh¡‹¥Hû bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP—¬ŠHü bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPw\@¥Jý bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP[ãJþ bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP—K?ÄJÿ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPøÊ&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.17761076986789703 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00912698358297348 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.026868902146816254 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.20445623993873596 particle/accuracy_2_of_5‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4734432101249695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.13710571825504303 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_3_of_5‚1 particle/accuracy_6_of_7‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12165984511375427 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.151785716 Å„ù4144516 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08444444835186005 jet/accuracy_6_of_7‚0 jet/accuracy_7_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9869778752326965 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.15080377459526062 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9407647848129272 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05273232236504555 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.03970205411314964 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9700083136558533 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 _runtime‚ 954.121770986 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1290278136730194 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9654001593589783 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9821926951408386 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.19377106428146362 +particle/sensitivity‚0.2568295896053314 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.37369585037231445 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3447522222995758 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.004202956333756447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9918981790542603 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9423747658729553 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9631782174110413 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2209904044866562 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19728748500347137 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3918651044368744 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9545454382896423 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11959131062030792 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_7_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.971034586429596 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4244786202907562 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05819933861494064 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05319417640566826 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07088122516870499 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1710936278104782 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11742424219846725 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03191489353775978 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1970842331647873 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2440476268529892 ,jet/accuracy_3_of_4‚0.036580536514520645 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.02658049948513508 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 epoch‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0033275627065449953 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.22813166677951813 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2916666865348816 *jet/accuracy_1_of_6‚0.9230769276618958 jet/accuracy_4_of_6‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1731448769569397 +particle/specificity‚0.8767750263214111 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.24044251441955566 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10796879231929779 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.20909342169761658 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9806148409843445 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 -jet/accuracy_4_of_4‚0.0029839712660759687 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9694459438323975 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.22828687727451324 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 jet/accuracy_1_of_7‚0.5 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4392601251602173 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11544924229383469 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20577655732631683 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9845858216285706 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16952930390834808 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.060847505927085876 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0357142873108387 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.21204371750354767 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.29933688044548035 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3645077049732208 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09113021194934845 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11395615339279175 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11816416680812836 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0589248351752758 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.17892754077911377 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9546327590942383 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.23221087455749512 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.22597406804561615 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9518849849700928 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4508128762245178 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11704272031784058 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.16042886674404144 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.06937166303396225 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚1208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 *jet/accuracy_1_of_4‚0.7483312487602234 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.2149449586868286 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.14715716242790222 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08852013945579529 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.11164020746946335 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.10725308954715729 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9686307907104492 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5530303120613098 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1736111044883728 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9610167741775513 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.023747514933347702 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.16688451170921326 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 particle/accuracy_2_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.21588604152202606 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25006404519081116 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.25675860047340393 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5020833015441895 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.12048610299825668 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.35293516516685486 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9915936589241028 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12710686028003693 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.17653508484363556 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.31578949093818665 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06851851940155029 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.7910451889038086 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2290765941143036 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.1975374072790146 particle/accuracy_4_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.04903090372681618 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9483067989349365 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17495130002498627 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2643481194972992 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40035539865493774 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09271540492773056 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.26327893137931824 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.183612659573555 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19586431980133057 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2963586747646332 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09263870120048523 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9874404668807983 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 % _timestamp‚1.7884174543822005e+09 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9712592363357544 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1028735563158989 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9707955718040466 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1671023964881897 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3832304775714874 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -jet/accuracy_3_of_3‚0.0055734277702867985 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.170138880610466 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.30324074625968933 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03440038487315178 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9700847268104553 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9818536043167114 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193103790283 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1173010841012001 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2133633941411972 ,jet/accuracy_3_of_5‚0.025089606642723083 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.22615590691566467 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.1958398073911667 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2332589477300644 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2839505970478058 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9364662766456604 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1975845992565155 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04386177286505699 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 +jet/accuracy_2_of_2‚0.05160675570368767 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9874604940414429 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12119093537330627 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2828975021839142 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 particle/accuracy_5_of_7‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9874371290206909 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.20781689882278442 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0034867534413933754 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6053987145423889 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9255208969116211 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010416865348816 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2259598970413208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3561151623725891 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 *jet/accuracy_1_of_5‚0.8279761672019958 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9563939571380615 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.241957888007164 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17495130002498627 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 (particle/accuracy‚0.7800149917602539 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9857499599456787 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08317926526069641 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.3125 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.607601523399353 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9658218622207642 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11219576001167297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08838096261024475 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.08144679665565491 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 _step‚98 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31470316648483276 )loss/val_total_loss‚67.46204547596567 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0025502536445856094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.17298851907253265 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11442424356937408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9644746780395508 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07407407462596893 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10822714865207672 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13669487833976746 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1408710479736328 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.015804599970579147 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3000449240207672 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015723271295428276 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13001839816570282 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_1_of_1‚0.30930033326148987 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9757707715034485 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.2944289445877075 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22000309824943542 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4330357611179352 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_1‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.292385071516037 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.05685001239180565 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.11695682257413864 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.37506240606307983 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1840304732322693 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.22423158586025238 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.19166666269302368 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.14259259402751923 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9033333659172058 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09517908096313477 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12162162363529205 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.272988498210907 particle/accuracy_1_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.1401515156030655 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.010187099687755108 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11754385381937027 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9852340221405029 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.17880676686763763 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.16763563454151154 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12654981017112732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5681818127632141 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.29333335161209106 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01036866381764412 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.20724827í´cV305116272 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23315875232219696 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11320546269416809 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07871690392494202 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9668532609939575 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05273232236504555 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9672413468360901 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9845858216285706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.681727945804596 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.5560795068740845 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.14230355620384216 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.14827586710453033 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0Ý——¿w› É¥ @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.31578949093818665 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.06851851940155029 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.7910451889038086 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2290765941143036 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.1975374072790146 particle/accuracy_4_of_4‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.04903090372681618 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9483067989349365 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.17495130002498627 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2643481194972992 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.40035539865493774 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09271540492773056 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.26327893137931824 particle/accuracy_1_of_2‚1 particle/accuracy_5_of_5‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.1666666567325592 8 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.02777777798473835 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.183612659573555 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.19586431980133057 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2963586747646332 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09263870120048523 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9874404668807983 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 % _timestamp‚1.7884174543822005e+09 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9712592363357544 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1028735563158989 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9707955718040466 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 jet/accuracy_4_of_7‚0 & Purity/3FRt*SRqqt2FBt/FRt_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.20000000298023224 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.1671023964881897 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3832304775714874 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 -jet/accuracy_3_of_3‚0.0055734277702867985 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.170138880610466 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.30324074625968933 + Purity/*FRt2SRqqt2FBt/FRt_purity‚0.1875 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03440038487315178 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9700847268104553 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9818536043167114 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193103790283 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1173010841012001 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.2133633941411972 ,jet/accuracy_3_of_5‚0.025089606642723083 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.22615590691566467 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.1958398073911667 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_3‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2332589477300644 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.2839505970478058 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.03846153989434242 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9364662766456604 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1975845992565155 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.04386177286505699 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 +jet/accuracy_2_of_2‚0.05160675570368767 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.03947368264198303 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9874604940414429 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12119093537330627 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.2828975021839142 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 particle/accuracy_5_of_7‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9874371290206909 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.20781689882278442 <"Purity/*FRt2SRqqt0FBt/event_purity‚0.0034867534413933754 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6053987145423889 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9255208969116211 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9010416865348816 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2259598970413208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3561151623725891 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 *jet/accuracy_1_of_5‚0.8279761672019958 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9563939571380615 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.241957888007164 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.17495130002498627 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 (particle/accuracy‚0.7800149917602539 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9857499599456787 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.08317926526069641 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.3125 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.607601523399353 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9658218622207642 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11219576001167297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08838096261024475 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.08144679665565491 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 _step‚98 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31470316648483276 )loss/val_total_loss‚67.46204547596567 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0025502536445856094 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.17298851907253265 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.11442424356937408 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9644746780395508 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07407407462596893 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.10822714865207672 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13669487833976746 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1408710479736328 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.015804599970579147 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3000449240207672 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.015723271295428276 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13001839816570282 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_1_of_1‚0.30930033326148987 particle/accuracy_1_of_3‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9757707715034485 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 6validation_average_jet_accuracy‚0.2944289445877075 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22000309824943542 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4330357611179352 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_1_of_1‚1 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.292385071516037 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.05685001239180565 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.11695682257413864 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.37506240606307983 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.1840304732322693 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.22423158586025238 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.19166666269302368 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.14259259402751923 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9033333659172058 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09517908096313477 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.12162162363529205 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.272988498210907 particle/accuracy_1_of_7‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.1401515156030655 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.010187099687755108 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11754385381937027 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9852340221405029 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.17880676686763763 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.16763563454151154 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8903509378433228 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12654981017112732 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5681818127632141 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.29333335161209106 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01036866381764412 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.02777777798473835 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2072482705116272 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.23315875232219696 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11320546269416809 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.07871690392494202 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9668532609939575 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8833333253860474 jet/accuracy_2_of_7‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05273232236504555 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9672413468360901 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9845858216285706 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.681727945804596 jet/accuracy_5_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.5560795068740845 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.14230355620384216 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.14827586710453033 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.18065975606441498 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9487679600715637 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.23148147761821747 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.008206960745155811 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4257444441318512 'particle/f_score‚0.2671073377132416 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt1FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.16398967802524567 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3323476016521454 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.17761076986789703 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00912698358297348 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.026868902146816254 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.20445623993873596 particle/accuracy_2_of_5‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4734432101249695 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.13710571825504303 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_3_of_5‚1 particle/accuracy_6_of_7‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12165984511375427 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1517857164144516 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08444444835186005 jet/accuracy_6_of_7‚0 jet/accuracy_7_of_7‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9869778752326965 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.15080377459526062 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9407647848129272 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05273232236504555 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.03970205411314964 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9700083136558533 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 _runtime‚ 954.121770986 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1290278136730194 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9654001593589783 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9821926951408386 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.19377106428146362 +particle/sensitivity‚0.2568295896053314 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.37369585037231445 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ) Purity/1FRt0SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3447522222995758 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.004202956333756447 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9918981790542603 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9423747658729553 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9631782174110413 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.2209904044866562 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.19728748500347137 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3918651044368744 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9545454382896423 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11959131062030792 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_7_of_7‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.971034586429596 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4244786202907562 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05819933861494064 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05319417640566826 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07088122516870499 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.2777777910232544 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.1710936278104782 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11742424219846725 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.20370370149612427 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.03191489353775978 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1970842331647873 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2440476268529892 ,jet/accuracy_3_of_4‚0.036580536514520645 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.02658049948513508 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 epoch‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0033275627065449953 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.22813166677951813 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2916666865348816 *jet/accuracy_1_of_6‚0.9230769276618958 jet/accuracy_4_of_6‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.1731448769569397 +particle/specificity‚0.8767750263214111 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.24044251441955566 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10796879231929779 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.20909342169761658 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9806148409843445 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9928571581840515 -jet/accuracy_4_of_4‚0.0029839712660759687 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4107142984867096 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9694459438323975 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.22828687727451324 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 jet/accuracy_1_of_7‚0.5 jet/accuracy_5_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8703703284263611 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4392601251602173 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.28947368264198303 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11544924229383469 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.20577655732631683 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.046875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity40ÐR‚0.75 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9845858216285706 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.16952930390834808 ;"Purity/*FRt*SRqqt0FBt/event_purity‚0.060847505927085876 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1597222238779068 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.0357142873108387 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.21204371750354767 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.29933688044548035 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.20000000298023224 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.39666667580604553 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9126984477043152 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3645077049732208 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.09113021194934845 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.11395615339279175 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11816416680812836 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0589248351752758 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.17892754077911377 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9546327590942383 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.23221087455749512 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.22597406804561615 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9518849849700928 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4508128762245178 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.11704272031784058 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.16042886674404144 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.06937166303396225 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 trainer/global_step‚1208 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 *jet/accuracy_1_of_4‚0.7483312487602234 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.2149449586868286 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.14715716242790222 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08852013945579529 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.11164020746946335 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.10725308954715729 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt*SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9686307907104492 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5530303120613098 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.1736111044883728 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9610167741775513 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.023747514933347702 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.16688451170921326 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 particle/accuracy_2_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.21588604152202606 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.25006404519081116 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.25675860047340393 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5020833015441895 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.12048610299825668 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.35293516516685486 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9915936589241028 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12710686028003693 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.17653508484363556 ("Purity/4FRt2SRqqt*FBt/event_purity‚0b³:¢œ ‚(:… ´«äÔÌ»‘~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚171.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.208 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÉ’£E ‚(:º ´«äÔÃèÑ% disk.nvme0n1.out‚3023.31591796875 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚16.99982855687447% disk./.usageGB‚253.30973052978516 disk.dm-0.in‚0.09375 disk.drbd1716.out‚0 disk.sda.out‚ 39.6328125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚0.09375 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.61328125 disk.nvme0c0n1.in‚0.09375' disk.nvme0c0n1.out‚3023.31591796875 disk.drbd2080.in‚0" disk.dm-0.out‚3023.31591796875 disk.sda.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 39.6328125 cpu‚25.36729612493208 network.sent‚ 1.595525e+06 network.recv‚425154 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 39.6328125 disk.sda3.in‚0$ memory_percent‚89.80569839477539' proc.memory.availableMB‚ 1670.234375# proc.memory.rssMB‚ 2765.51953125) proc.memory.percent‚16.87939167022705€3ÐÃœž ‚(:‘ ëä﵆ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚139.72' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.944000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö0{HŸ ‚(:½ ëäÔ®™Ÿ disk.rbd1.in‚0 disk.sda2.out‚0% disk./.usageGB‚253.31829452514648 disk.drbd2266.out‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0" disk.dm-0.out‚3990.30029296875 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚95.59767246246338 disk.dm-0.in‚0.09375 disk.rbd0.in‚0 network.recv‚432343 disk.dm-1.out‚ 39.9921875 disk.rbd1.out‚ 0.61328125% disk.nvme0n1.out‚3990.30029296875' disk.nvme0c0n1.out‚3990.30029296875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚0.09375* proc.memory.percent‚16.879677772521973 disk.dm-1.in‚0 disk.sda1.in‚0 cpu‚25.362562070731627 network.sent‚ 1.720632e+06 disk.sda.out‚ 39.9921875 disk.sda3.out‚ 39.9921875( proc.memory.availableMB‚ 721.27734375# proc.memory.rssMB‚ 2765.56640625) disk./.usagePercent‚17.00040329378884 disk.drbd2080.out‚0Gsظ0  + _wandbruntime‚987  _runtime‚987½½Ç¡ Á ' lr-AdamW/pg1‚0.00016868098209987957 ' lr-AdamW/pg2‚0.00016868098209987957 trainer/global_step‚1249 % _timestamp‚1.7884174878420606e+09 _runtime‚ 987.579634885 _step‚99ƒ“ZôË¢ Å ' lr-AdamW/pg1‚0.00016868098209987957 ' lr-AdamW/pg2‚0.00016868098209987957 trainer/global_step‚1249 % _timestamp‚1.7884174878420606e+09 _runtime‚ 987.579634885 _step‚99cènS0£ + _wandbruntime‚987  _runtime‚987Ö¢eÆ'¤ ÀO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18208376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚70869370475520 trainer/global_step‚1249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18208374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72778502860288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80571816439808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 987.580484759 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79470224662016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚100 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884174878427787e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16701046 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71912909329803 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73826573425152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚73825899060736 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1048070564864 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16701474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1043555893643 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚73826573425152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10625454 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6075594 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80571635166208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 987.580484759 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71913580806911 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23607684 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72778484857344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79470224662016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1044210331391 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚100 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10625452 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23627720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18208374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1101410504192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1101567837696 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70869353436160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884174878427787e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41816058 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41836096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16701046 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71912909329803 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚73826573425152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23627720 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72778502860288 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72778502860288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6076020 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1047414203392 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23607684 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80571816439808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1048070564864 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038dÅŸÇ0¦ + _wandbruntime‚988  _runtime‚988× Ä)§ ÊM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884174885271983e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161514496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776ÖT#cÔ&¨ ÎM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23646200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23626138 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80634332342272 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41848528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18222390 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1102433002496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚79531899339776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10633620 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41868592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚73883745905152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70924274606080 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18222392 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72834837896192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41868592 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18222390 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6080519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72834837896192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1044372328607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1249 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚73883745905152 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884174885271983e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161514496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18222392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16714141 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚438 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41848528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16713703 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71969301372643 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944eפ«ÿ0© + _wandbruntime‚988  _runtime‚988•î$µRª Ì .loss/FRt4/detection_loss‚2.639463214871456 /loss/FBt4/detection_loss‚1.3825575473019853 1loss/FBt4/assignment_loss‚0.20067147153527287 .loss/FBt1/detection_loss‚5.783242081302436 trainer/global_step‚1249 $ _timestamp‚1.788417488529406e+09 _runtime‚ 988.26686962 _step‚102 1loss/SRqqt1/assignment_loss‚2.900461175612041 .loss/FRt1/detection_loss‚2.376338608124677 /loss/FRt3/detection_loss‚2.6392840553413737 1loss/SRqqt1/detection_loss‚2.8496956718819484 0loss/SRqqt2/detection_loss‚2.271351627111435 /loss/FRt1/assignment_loss‚6.457893287434297 0loss/FRt3/assignment_loss‚5.6621283076026225 2loss/SRqqt4/assignment_loss‚1.8735123375406513 0loss/FBt1/assignment_loss‚0.5766879098955542 /loss/FRt2/detection_loss‚2.3051913944157687 %loss/total_loss‚71.59616828148415 1loss/SRqqt3/assignment_loss‚2.342150568962097 0loss/FBt3/assignment_loss‚0.7588144806213677 2loss/SRqqt2/assignment_loss‚2.7981759548187255 1loss/SRqqt3/detection_loss‚3.3103315798299655 .loss/FBt2/detection_loss‚6.532266162335873 .loss/FBt3/detection_loss‚5.361507885158062 epoch‚1 .loss/FRt2/assignment_loss‚4.89002508466894 0loss/FRt4/assignment_loss‚3.6325064661367885 0loss/FBt2/assignment_loss‚0.6161879608025629 1loss/SRqqt4/detection_loss‚1.4357234481782593õЗV« Ð .loss/FRt4/detection_loss‚2.639463214871456 /loss/FBt4/detection_loss‚1.3825575473019853 1loss/FBt4/assignment_loss‚0.20067147153527287 .loss/FBt1/detection_loss‚5.783242081302436 trainer/global_step‚1249 $ _timestamp‚1.788417488529406e+09 _runtime‚ 988.26686962 _step‚102 1loss/SRqqt1/assignment_loss‚2.900461175612041 .loss/FRt1/detection_loss‚2.376338608124677 /loss/FRt3/detection_loss‚2.6392840553413737 1loss/SRqqt1/detection_loss‚2.8496956718819484 0loss/SRqqt2/detection_loss‚2.271351627111435 /loss/FRt1/assignment_loss‚6.457893287434297 0loss/FRt3/assignment_loss‚5.6621283076026225 2loss/SRqqt4/assignment_loss‚1.8735123375406513 0loss/FBt1/assignment_loss‚0.5766879098955542 /loss/FRt2/detection_loss‚2.3051913944157687 %loss/total_loss‚71.59616828148415 1loss/SRqqt3/assignment_loss‚2.342150568962097 0loss/FBt3/assignment_loss‚0.7588144806213677 2loss/SRqqt2/assignment_loss‚2.7981759548187255 1loss/SRqqt3/detection_loss‚3.3103315798299655 .loss/FBt2/detection_loss‚6.532266162335873 .loss/FBt3/detection_loss‚5.361507885158062 epoch‚1 .loss/FRt2/assignment_loss‚4.89002508466894 0loss/FRt4/assignment_loss‚3.6325064661367885 0loss/FBt2/assignment_loss‚0.6161879608025629 1loss/SRqqt4/detection_loss‚1.4357234481782593fËÛÇì¬ ‚(:’ Ò«äÔ‘³~ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѹ»dE­ ‚(:º Ò«äÔúÏØ€ disk.dm-1.out‚ 40.60546875 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 cpu‚25.35773080407343 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚95.59977054595947( proc.memory.availableMB‚ 720.93359375 network.recv‚436818* disk./.usagePercent‚17.001047152737453% disk./.usageGB‚253.32788848876953 disk.drbd2080.in‚0 network.sent‚ 1.724431e+06% disk.nvme0c0n1.out‚4034.666015625 disk.sda3.in‚0 disk.sda3.out‚ 40.60546875 disk.drbd2265.in‚0 disk.dm-0.out‚4034.666015625 disk.dm-1.in‚0 disk.sda.out‚ 40.60546875 disk.nvme0n1.in‚0.09375# disk.nvme0n1.out‚4034.666015625 disk.drbd2265.out‚0 disk.dm-0.in‚0.09375 disk.sda1.out‚0# proc.memory.rssMB‚ 2765.56640625 disk.drbd1716.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.879677772521973 disk.nvme0c0n1.in‚0.09375 disk.drbd2080.out‚0 disk.sda.in‚0ƒÅ#’® ‚(:‡ á«äÔÂà¬~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÚþPG¯ ‚(:¼ á«äÔ¸°€! disk.dm-0.out‚4077.1357421875 disk.rbd1.out‚ 0.61328125$ disk.nvme0n1.out‚4077.1357421875 disk.sda2.out‚0 disk.drbd2080.in‚0 cpu‚25.353270426951973 network.recv‚445204 disk.drbd1716.in‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0$ memory_percent‚95.60003280639648 disk.sda.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 41.23828125 proc.cpu.threads‚17* disk./.usagePercent‚17.001736581027163 disk.sda1.in‚0 disk.nvme0c0n1.in‚0.09375& disk.nvme0c0n1.out‚4077.1357421875* proc.memory.percent‚16.879677772521973 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0.09375 disk.sda2.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2765.56640625 network.sent‚ 1.832819e+06 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 720.890625% disk./.usageGB‚253.33816146850586 disk.sda.out‚ 41.23828125 disk.dm-1.out‚ 41.23828125>£Œš’° ‚(:‡ ð«äÔ¹¢£~ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»W1q?± ‚(:´ ð«äÔ寚 disk.drbd2080.in‚0 disk.sda.in‚0 cpu‚25.34888065575574 proc.cpu.threads‚17 network.sent‚ 1.839285e+06 disk.drbd1716.out‚0 disk.nvme0n1.in‚0.09375# disk.nvme0n1.out‚4121.630859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 network.recv‚450151 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 41.6953125$ memory_percent‚95.60155868530273& proc.memory.availableMB‚ 720.640625# proc.memory.rssMB‚ 2765.56640625% disk.nvme0c0n1.out‚4121.630859375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda3.in‚0 disk.dm-0.out‚4121.630859375 disk.dm-1.out‚ 41.6953125 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.879677772521973$ disk./.usageGB‚253.3445587158203 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 41.6953125 disk.rbd1.out‚ 0.61328125* disk./.usagePercent‚17.002165905662086 disk.sda2.out‚0 disk.dm-0.in‚0.09375”„Ë2² - _wandbruntime‚1023  _runtime‚1023é\ú‹È³  trainer/global_step‚1299 $ _timestamp‚1.788417524245197e+09 _runtime‚1023.982727952 _step‚103 ' lr-AdamW/pg1‚0.00016865813311933706 ' lr-AdamW/pg2‚0.00016865813311933706÷žpÌ´ Æ ' lr-AdamW/pg1‚0.00016865813311933706 ' lr-AdamW/pg2‚0.00016865813311933706 trainer/global_step‚1299 $ _timestamp‚1.788417524245197e+09 _runtime‚1023.982727952 _step‚103gýUîÑ2µ - _wandbruntime‚1023  _runtime‚1023x×͹¶ ÁO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1089056370688 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75596136799744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6302438 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74699622674319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75596154802688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75596136799744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76685211173376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82552761545216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚1299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24533736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76685211173376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocatgšbì ion.small_pool.current‚20036 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76685211173376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76684536808960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75596154802688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75596136799744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18909174 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚104 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76685211173376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82552761545216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚1299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24533736hO’‚Ð2¸ - _wandbruntime‚1024  _runtime‚1024 Ï$Ï&¹ ÉM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43475380 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1089875811840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18923192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 _runtime‚1024.669193051 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76741709275648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24552190 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83759570876928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6306917 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183357440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24572252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11042020 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1145158594048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¹¹$º ÍM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74754671762735 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76742383653376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18923190 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83759754234368 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43495444 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1145318011392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11042022 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1145158594048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1085862105971 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43475380 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159417344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17348513 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚75652507841536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1089875811840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18923192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18923192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43495444 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82614412282880 _runtime‚1024.669193051 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76741709275648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1085207668015 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24552190 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74755343240051 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1089219437056 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83759570876928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18923190 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1089219437056 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6306917 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183357440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24572252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11042020 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 @_½·ÐO¿ ‚(:³ ÿ«äÔµêž disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.34488920075575 disk.sda.out‚ 42.171875 disk.nvme0n1.in‚0.09375 disk.dm-1.out‚ 42.171875 proc.cpu.threads‚17# disk.nvme0n1.out‚4159.650390625 disk.sda2.in‚0# proc.memory.rssMB‚ 2765.56640625 network.recv‚453697 disk.drbd2266.out‚0 disk.dm-0.out‚4159.650390625$ memory_percent‚95.59972286224365( proc.memory.availableMB‚ 720.94140625 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0.09375 network.sent‚ 1.844585e+06 disk.drbd1716.out‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 42.171875 disk.drbd2080.out‚0) disk./.usagePercent‚17.00310263722987 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0% disk.nvme0c0n1.out‚4159.650390625 disk.sda2.out‚0% disk./.usageGB‚253.35851669311523 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.09375* proc.memory.percent‚16.879677772521973 disk.rbd1.in‚0¢]q÷‘À ‚(:† Ž¬äÔÑê¹~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îqIÁ ‚(:¾ Ž¬äÔƒŸ¶€ disk.sda1.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.drbd1716.out‚0* proc.memory.percent‚16.879677772521973 disk.sda.out‚ 42.6953125 disk.sda1.out‚0% disk.nvme0n1.out‚4207.97216796875' disk.nvme0c0n1.out‚4207.97216796875 disk.sda2.in‚0( proc.memory.availableMB‚ 720.87890625 network.recv‚460346* disk./.usagePercent‚17.003549626384654 disk.sda.in‚0 disk.dm-1.out‚ 42.6953125 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.sent‚ 1.952012e+06 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚0.09375$ memory_percent‚95.60010433197021% disk./.usageGB‚253.36517715454102 disk.dm-1.in‚0 disk.sda3.out‚ 42.6953125# proc.memory.rssMB‚ 2765.56640625 disk.rbd1.in‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚0.09375" disk.dm-0.out‚4207.97216796875 cpu‚25.3408189095503968Plö2 - _wandbruntime‚1060  _runtime‚1060ñfâÉà à _runtime‚1060.650494757 _step‚107 ' lr-AdamW/pg1‚0.00016861771305465306 ' lr-AdamW/pg2‚0.00016861771305465306 trainer/global_step‚1349 % _timestamp‚1.7884175609129121e+09XÒ‡ÍÄ Ç _runtime‚1060.650494757 _step‚107 ' lr-AdamW/pg1‚0.00016861771305465306 ' lr-AdamW/pg2‚0.00016861771305465306 trainer/global_step‚1349 % _timestamp‚1.7884175609129121e+09kÑì2Å - _wandbruntime‚1060  _runtime‚1060mÀLmÆ'Æ ÀO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76359783531520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1129385815040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25479824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6529308 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77484993064619 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17971989 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19609974 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1130042176512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1125881010207 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25479824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19609976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25459788 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78413806745088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11442254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45069762 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86822591273984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85635298428416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1187268905472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25459788 trainer/global_step‚1349 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79543848921600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76359766492160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 _runtime‚1060.651378725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1129385815040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77485664541727 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19609976 _step‚108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6529735 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1130042176512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11442254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45069762 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86822591273984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1187113669120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85635298428416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1187268905472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79543848921600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79543174557184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19609974 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25459788 trainer/global_step‚1349 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79543174557184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78413788742144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45089800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179176448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79543848921600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155236352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17971560 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76359766492160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85635322368512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1125226572459 _runtime‚1060.651378725 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1129385815040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11442252 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77485664541727 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78413788742144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19609976 _step‚108 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6529735 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚1349 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19623990 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85696949166080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1187974833664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1126697445379 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1126043007423 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all\ötS.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19623992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79601021401600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚79601021401600 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19623992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79601021401600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1130205242880 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17984248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85696973106176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1188138445312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79600347023872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25498304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76414687662080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45122296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11450422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚1349 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25498304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78470141780992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19623992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25478242 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0må³Ê2Ë - _wandbruntime‚1061  _runtime‚1061>ÛæSÌ Í /loss/FBt3/assignment_loss‚0.561036035031672 /loss/FRt3/detection_loss‚3.0239051912512096 1loss/SRqqt1/detection_loss‚3.0780962911935954 .loss/FBt2/detection_loss‚5.287394489560809 2loss/SRqqt1/assignment_loss‚2.8281957575908074 1loss/SRqqt3/detection_loss‚2.3658604403821433 /loss/FBt1/detection_loss‚2.5925859238910767 _runtime‚ 1061.35516417 $loss/total_loss‚62.8089180228857 1loss/FBt2/assignment_loss‚0.46708705700335224 .loss/FRt1/detection_loss‚2.644005561697072 /loss/FRt2/detection_loss‚2.3857235779177466 /loss/FRt1/assignment_loss‚6.343074288861505 0loss/FRt3/assignment_loss‚4.8042107037135535 0loss/FRt4/assignment_loss‚3.1774343273856425 2loss/SRqqt2/assignment_loss‚2.0038755156245887 /loss/FBt4/assignment_loss‚0.263356365330027 1loss/SRqqt4/detection_loss‚1.2765984830336385 epoch‚1 2loss/SRqqt4/assignment_loss‚1.6956371610523553 0loss/FBt1/assignment_loss‚0.5054307638012246 .loss/FBt3/detection_loss‚4.459867590003544 % _timestamp‚1.7884175616177146e+09 _step‚110 .loss/FBt4/detection_loss‚2.128470433441301 trainer/global_step‚1349 .loss/FRt2/assignment_loss‚4.37677703713471 .loss/FRt4/detection_loss‚2.315494224970991 1loss/SRqqt2/detection_loss‚2.1271148604505203 1loss/SRqqt3/assignment_loss‚2.097685942562615^ÈìÆWÍ Ñ /loss/FBt1/detection_loss‚2.5925859238910767 _runtime‚ 1061.35516417 $loss/total_loss‚62.8089180228857 1loss/FBt2/assignment_loss‚0.46708705700335224 .loss/FRt1/detection_loss‚2.644005561697072 /loss/FRt2/detection_loss‚2.3857235779177466 /loss/FRt1/assignment_loss‚6.343074288861505 0loss/FRt3/assignment_loss‚4.8042107037135535 0loss/FRt4/assignment_loss‚3.1774343273856425 2loss/SRqqt2/assignment_loss‚2.0038755156245887 /loss/FBt4/assignment_loss‚0.263356365330027 1loss/SRqqt4/detection_loss‚1.2765984830336385 epoch‚1 2loss/SRqqt4/assignment_loss‚1.6956371610523553 0loss/FBt1/assignment_loss‚0.5054307638012246 .loss/FBt3/detection_loss‚4.459867590003544 % _timestamp‚1.7884175616177146e+09 _step‚110 .loss/FBt4/detection_loss‚2.128470433441301 trainer/global_step‚1349 .loss/FRt2/assignment_loss‚4.37677703713471 .loss/FRt4/detection_loss‚2.315494224970991 1loss/SRqqt2/detection_loss‚2.1271148604505203 1loss/SRqqt3/assignment_loss‚2.097685942562615 /loss/FBt3/assignment_loss‚0.561036035031672 /loss/FRt3/detection_loss‚3.0239051912512096 1loss/SRqqt1/detection_loss‚3.0780962911935954 .loss/FBt2/detection_loss‚5.287394489560809 2loss/SRqqt1/assignment_loss‚2.8281957575908074 1loss/SRqqt3/detection_loss‚2.3658604403821433n5¨ÉªÎ ‚(:’ ¬äÔµ÷†~ gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.167599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ç~ÂJÏ ‚(:¿ ¬äÔß“˜€ disk.drbd2265.out‚0 disk.sda.out‚ 43.51953125 disk.sda1.out‚0% disk.nvme0n1.out‚4259.08154296875 network.recv‚464471$ disk./.usageGB‚253.4233055114746 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 43.51953125 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚0.09375' disk.nvme0c0n1.out‚4259.08154296875 disk.sda3.out‚ 43.51953125 disk.dm-0.in‚0.09375 disk.rbd1.out‚ 0.61328125 disk.drbd1716.in‚0 disk.rbd0.out‚0 cpu‚25.336832431568272 disk.rbd0.in‚0 disk.drbd2265.in‚0" disk.dm-0.out‚4259.08154296875 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.60189247131348 network.sent‚ 1.957912e+06 disk.rbd1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 720.5859375* proc.memory.percent‚16.879677772521973* disk./.usagePercent‚17.007450669191407 disk.drbd2080.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2765.56640625 disk.sda.in‚0 disk.nvme0c0n1.in‚0.09375 disk.sda2.out‚0 proc.cpu.threads‚17…Ó`±Ð ‚(:’ ¬¬äÔÇ ¸~ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.701' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.740199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZºDÑ ‚(:¹ ¬¬äÔËÍœ network.recv‚471880 disk.sda2.out‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 720.6328125 network.sent‚ 2.06655e+06$ memory_percent‚95.60160636901855 disk.drbd1716.out‚0* disk./.usagePercent‚17.009047132174928& disk.nvme0c0n1.out‚4319.4638671875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 43.8828125 disk.nvme0c0n1.in‚0.09375 disk.dm-1.out‚ 43.8828125 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2765.56640625* proc.memory.percent‚16.879677772521973 cpu‚25.332893820130142 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚253.44709396362305 disk.dm-0.in‚0.09375 disk.sda1.out‚0$ disk.nvme0n1.out‚4319.4638671875 disk.sda2.in‚0 disk.sda3.out‚ 43.8828125! disk.dm-0.out‚4319.4638671875 disk.rbd1.out‚ 0.61328125 disk.rbd1.in‚0 disk.sda1.in‚0æªiïœÒ ‚(:‘ »¬äÔøè­~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.989999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{»Ôp=Ó ‚(:² »¬äÔӮÀ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 720.578125 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚4363.95654296875 network.recv‚475595 disk.sda1.out‚0 disk.sda3.out‚44.4375 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375* proc.memory.percent‚16.879677772521973 cpu‚25.329141234615015' disk.nvme0c0n1.out‚4363.95654296875% disk./.usageGB‚253.45153427124023 disk.drbd1716.out‚0% disk.nvme0n1.out‚4363.95654296875 disk.rbd0.out‚0 disk.rbd1.out‚ 0.61328125 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.dm-1.out‚44.4375 disk.sda.out‚44.4375 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2765.56640625 proc.cpu.threads‚17 disk.sda.in‚0# memory_percent‚95.6019401550293 disk.nvme0c0n1.in‚0.09375 network.sent‚ 2.071864e+06* disk./.usagePercent‚17.009345124944783|ÂLO2Ô - _wandbruntime‚1097  _runtime‚1097¿ŒnµÉÕ Ã _step‚111 ' lr-AdamW/pg1‚0.00016855973032930857 ' lr-AdamW/pg2‚0.00016855973032930857 trainer/global_step‚1399 % _timestamp‚1.7884175977138038e+09 _runtime‚1097.451356537K£v¨ÍÖ Ç % _timestamp‚1.7884175977138038e+09 _runtime‚1097.451356537 _step‚111 ' lr-AdamW/pg1‚0.00016855973032930857 ' lr-AdamW/pg2‚0.00016855973032930857 trainer/global_step‚1399oµÕ2× - _wandbruntime‚1097  _runtime‚1097§ SKÇ'Ø ÁO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88717859251712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1097.452205108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18606816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80271706409135 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1170371620864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89947786724864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82402486669824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚79104973020160 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26405876 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89947974289920 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82402486669824 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81231440684544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1171027982336 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20310774 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1229951413248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1399 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20310774 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11850654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1166716349615 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6756164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚112DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1229951413248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1399 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79104990059520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20310774 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82401812305408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81231458687488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11850654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1166716349615 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1171027982336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80271034932027 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1170371620864 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6756164 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1230115038208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20310776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187565056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88717859251712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚1097.452205108 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46696614 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18606816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80271706409135 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81231440684544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26405876 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1170371620864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6756609 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26385840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89947786724864 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18607263p:,¹—2Ú - _wandbruntime‚1098  _runtime‚1098°¹Ö—Ð&Û ÊM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26404294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90010283329024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88779509989376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20324790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88779486049280 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80327426974867 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82459659149824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884175984178371e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728z£,Ô&Ü ÎM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81287793723392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚79159894190080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82459659149824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46729084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26424356 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80326755497551 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26404294 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1167532784787 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884175984178371e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚79159877150720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20324792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1171191048704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1171191048704 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11858822 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1166878346831 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26404294 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20324792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6761074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90010283329024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1230969280000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46749148 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26424356 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88779509989376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18619451 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20324790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82458984772096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88779486049280 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6760631 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46729084 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46749148 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80327426974867 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1171847423488 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18619896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1171847423488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90010479269376 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81287793723392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844qÁíÓ2Ý - _wandbruntime‚1098  _runtime‚1098µ-â°SÞ Í /loss/FRt1/assignment_loss‚5.730977701104206 /loss/FRt2/assignment_loss‚4.154276122462075 /loss/FRt3/detection_loss‚3.0575439430052236 0loss/SRqqt1/detection_loss‚2.992802019119263 .loss/FBt3/detection_loss‚4.808829528944833 .loss/FBt4/detection_loss‚1.632029544155706 2loss/SRqqt4/assignment_loss‚1.6715166476625267 /loss/FRt4/detection_loss‚2.0145570028383855 epoch‚1 /loss/FRt2/detection_loss‚2.1932506769333244 ,loss/FBt2/detection_loss‚9.7473304271698 .loss/FRt3/assignment_loss‚4.88336609255883 2loss/SRqqt1/assignment_loss‚2.9726273727416994 1loss/FBt1/assignment_loss‚0.37778634114825904 1loss/FBt4/assignment_loss‚0.18160785952505565 .loss/FBt1/detection_loss‚2.508566512749819 $ _timestamp‚1.788417598419457e+09 1loss/SRqqt3/assignment_loss‚2.474517203039593 0loss/FBt3/assignment_loss‚0.7105238947063169 /loss/FRt1/detection_loss‚2.8896727302800054 1loss/SRqqt2/detection_loss‚1.9910062688295957 1loss/SRqqt4/detection_loss‚1.2008408694578114 %loss/total_loss‚67.28217724134953 trainer/global_step‚1399 0loss/FBt2/assignment_loss‚1.1313981711864471 _runtime‚1098.156957703 _step‚114 /loss/FRt4/assignment_loss‚3.217779473733094 2loss/SRqqt2/assignment_loss‚2.1053794607764384 1loss/SRqqt3/detection_loss‚2.6339913772212133»3WSÌß Ñ /loss/FRt1/assignment_loss‚5.730977701104206 /loss/FRt2/assignment_loss‚4.154276122462075 /loss/FRt3/detection_loss‚3.0575439430052236 0loss/SRqqt1/detection_loss‚2.992802019119263 .loss/FBt3/detection_loss‚4.808829528944833 .loss/FBt4/detection_loss‚1.632029544155706 2loss/SRqqt4/assignment_loss‚1.6715166476625267 /loss/FRt4/detection_loss‚2.0145570028383855 epoch‚1 /loss/FRt2/detection_loss‚2.1932506769333244 ,loss/FBt2/detection_loss‚9.7473304271698 .loss/FRt3/assignment_loss‚4.88336609255883 2loss/SRqqt1/assignment_loss‚2.9726273727416994 1loss/FBt1/assignment_loss‚0.37778634114825904 1loss/FBt4/assignment_loss‚0.18160785952505565 .loss/FBt1/detection_loss‚2.508566512749819 $ _timestamp‚1.788417598419457e+09 1loss/SRqqt3/assignment_loss‚2.474517203039593 0loss/FBt3/assignment_loss‚0.7105238947063169 /loss/FRt1/detection_loss‚2.8896727302800054 1loss/SRqqt2/detection_loss‚1.9910062688295957 1loss/SRqqt4/detection_loss‚1.2008408694578114 %loss/total_loss‚67.28217724134953 trainer/global_step‚1399 0loss/FBt2/assignment_loss‚1.1313981711864471 _runtime‚1098.156957703 _step‚114 /loss/FRt4/ass”uë4‹ignment_loss‚3.217779473733094 2loss/SRqqt2/assignment_loss‚2.1053794607764384 1loss/SRqqt3/detection_loss‚2.6339913772212133rÌï~œà ‚(:‘ ʬäÔ窌 gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.33' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.666000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë¢ÿŸ7á ‚(:¬ ʬäÔœ›¿€) disk./.usagePercent‚17.01087476996187& disk.nvme0c0n1.out‚4427.6240234375 disk.sda3.out‚44.875 disk.dm-1.out‚44.875 cpu‚25.32545733615401 disk.sda2.out‚0 network.recv‚480160 disk.sda.in‚0 disk.sda.out‚44.875$ disk.nvme0n1.out‚4427.6240234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚0.09375 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚0.09375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2765.5703125* proc.memory.percent‚16.879701614379883 network.sent‚ 2.077851e+06 disk.drbd1716.in‚0! disk.dm-0.out‚4427.6240234375% disk./.usageGB‚253.47432708740234 disk.rbd1.out‚ 0.61328125 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0$ memory_percent‚95.60329914093018( proc.memory.availableMB‚ 720.35546875 proc.cpu.threads‚17Áó|Æâ ‚(:’ Ù¬äÔ”†þ} gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ɳHã ‚(:½ Ù¬äÔÚ¶„€$ memory_percent‚95.60339450836182 disk.sda3.out‚ 45.38671875 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚0.09375 disk.dm-1.out‚ 45.38671875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda.out‚ 45.38671875( proc.memory.availableMB‚ 720.33984375 cpu‚25.321671916395815 disk.sda1.in‚0 disk.nvme0n1.in‚0.09375* proc.memory.percent‚16.879701614379883* disk./.usagePercent‚17.012505793962514 disk.sda1.out‚0& disk.nvme0c0n1.out‚4470.6669921875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.sent‚ 2.186181e+06 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0$ disk.nvme0n1.out‚4470.6669921875! disk.dm-0.out‚4470.6669921875% disk./.usageGB‚253.49863052368164 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚0.09375" proc.memory.rssMB‚ 2765.5703125 network.recv‚487335 disk.sda2.in‚0 disk.drbd2080.in‚0ŒaNY2ä - _wandbruntime‚1133  _runtime‚1133?g6 Èå  trainer/global_step‚1449 % _timestamp‚1.7884176342552152e+09 _runtime‚ 1133.99274438 _step‚115 ' lr-AdamW/pg1‚0.00016848419702681657 ' lr-AdamW/pg2‚0.00016848419702681657J."Ìæ Æ _runtime‚ 1133.99274438 _step‚115 ' lr-AdamW/pg1‚0.00016848419702681657 ' lr-AdamW/pg2‚0.00016848419702681657 trainer/global_step‚1449 % _timestamp‚1.7884176342552152e+09sø×2ç - _wandbruntime‚1133  _runtime‚1133ƒcÕÆ'è ÀO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27331928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85261124418048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81850179548160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81850196587520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038†u‹Ê'é ÄO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85260450053632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91800396134912 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21011576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85260450053632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81850179548160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91800372194816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81850196587520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1207551689023 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48323466 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19244049 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27331928 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85261124418048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84049110629888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1212013788160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83057076799435 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27311892 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1273088425472 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84049110629888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1211357426688 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6984997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27331928 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84049092626944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19244485 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚85261124418048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93073484560384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚117 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83113468842275 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚1449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1212833229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27350408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 TFDevice[á_Æ)StatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91862022932480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1134.683360331Ã%nÕ&ì ÏM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19256757 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1208368124195 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1212176854528 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84105463668736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48376000 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚117 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84105445665792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83113468842275 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81905083678720 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21025592 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚1449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12267220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27330346 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48376000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1212833229312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27350408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84105445665792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19257199 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚91862046872576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93135986027520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83112797364959 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91862022932480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21025592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27350408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6989977 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1212833229312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1207713686239 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1134.683360331 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84105463668736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85318296898048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0u±Ñ2í - _wandbruntime‚1134  _runtime‚1134Ѭ'Tî Î 2loss/SRqqt3/assignment_loss‚2.3246526265585863 1loss/SRqqt4/assignment_loss‚1.629485506284982 0loss/FBt2/assignment_loss‚0.6557302375634512 0loss/SRqqt4/detection_loss‚1.582480842951918 /loss/FBt3/detection_loss‚3.7599625322553845 0loss/FBt1/assignment_loss‚0.6357287133739604 1loss/FBt4/assignment_loss‚0.26454643288356977 _step‚118 /loss/FRt1/assignment_loss‚5.342621630635755 /loss/FRt2/assignment_loss‚4.638991643400753 1loss/SRqqt1/assignment_loss‚2.663232195835847 2loss/SRqqt2/assignment_loss‚2.3277280787204173 .loss/FRt4/detection_loss‚2.224077207518035 .loss/FBt2/detection_loss‚6.664679368336995 /loss/FRt3/assignment_loss‚5.411983042955399 .loss/FRt1/detection_loss‚2.753717849994528 %loss/total_loss‚62.83725744464353 1loss/SRqqt1/detection_loss‚2.9817431580561857 1loss/SRqqt2/detection_loss‚2.2521637646441763 /loss/FBt4/detection_loss‚0.9611656396235857 0loss/SRqqt3/detection_loss‚2.972993250246401 /loss/FBt1/detection_loss‚1.8345688496582684 epoch‚1 trainer/global_step‚1449 _runtime‚1134.684357152 % _timestamp‚1.7884176349469297e+09 /loss/FRt4/assignment_loss‚2.661276887203085 0loss/FBt3/assignment_loss‚0.9305863285228876 /loss/FRt2/detection_loss‚2.4563590545280305 .loss/FRt3/detection_loss‚2.906782602891326 rÈ€Xï Ò 1loss/SRqqt2/detection_loss‚2.2521637646441763 /loss/FBt4/detection_loss‚0.9611656396235857 0loss/SRqqt3/detection_loss‚2.972993250246401 /loss/FBt1/detection_loss‚1.8345688496582684 epoch‚1 trainer/global_step‚1449 _runtime‚1134.684357152 % _timestamp‚1.7884176349469297e+09 /loss/FRt4/assignment_loss‚2.661276887203085 0loss/FBt3/assignment_loss‚0.9305863285228876 /loss/FRt2/detection_loss‚2.4563590545280305 .loss/FRt3/detection_loss‚2.906782602891326 2loss/SRqqt3/assignment_loss‚2.3246526265585863 1loss/SRqqt4/assignment_loss‚1.629485506284982 0loss/FBt2/assignment_loss‚0.6557302375634512 0loss/SRqqt4/detection_loss‚1.582480842951918 /loss/FBt3/detection_loss‚3.7599625322553845 0loss/FBt1/assignment_loss‚0.6357287133739604 1loss/FBt4/assignment_loss‚0.26454643288356977 _step‚118 /loss/FRt1/assignment_loss‚5.342621630635755 /loss/FRt2/assignment_loss‚4.638991643400753 1loss/SRqqt1/assignment_loss‚2.663232195835847 2loss/SRqqt2/assignment_loss‚2.3277280787204173 .loss/FRt4/detection_loss‚2.224077207518035 .loss/FBt2/detection_loss‚6.664679368336995 /loss/FRt3/assignment_loss‚5.411983042955399 .loss/FRt1/detection_loss‚2.753717849994528 %loss/total_loss‚62.83725744464353 1loss/SRqqt1/detection_loss‚2.9817431580561857v¥Üœ‘ð ‚(:† è¬äÔÐÚ± gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(eõDñ ‚(:¹ è¬äÔ»Õ©€ disk.sda.out‚ 46.0234375 network.sent‚ 2.192558e+06 disk.sda3.out‚ 46.0234375 disk.dm-1.out‚ 46.0234375 disk.sda.in‚0% disk./.usageGB‚253.54494857788086 disk.rbd1.in‚0" proc.memory.rssMB‚ 2765.5703125 cpu‚25.31816426182338 proc.cpu.threads‚17) disk./.usagePercent‚17.01561423744326 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚4523.51904296875 disk.drbd2265.out‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 720.296875 disk.sda3.in‚0" disk.dm-0.out‚4523.51904296875 disk.dm-1.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.879701614379883 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0.09375 disk.nvme0n1.in‚0.09375% disk.nvme0n1.out‚4523.51904296875 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚95.60365676879883 network.recv‚492153 disk.drbd1716.in‚0k?“ò ‚(:’ ÷¬äÔÖº€~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì°ª.Aó ‚(:¶ ÷¬äÔÂÓú€ disk.rbd0.out‚0( proc.memory.availableMB‚ 720.29296875 disk.drbd2265.out‚0 network.sent‚ 2.300145e+06$ disk./.usageGB‚253.5609359741211 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚4564.82763671875$ memory_percent‚95.60368061065674 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚0.09375 disk.nvme0c0n1.in‚0.09375 disk.drbd2080.out‚0* proc.memory.percent‚16.879701614379883 proc.cpu.threads‚17* disk./.usagePercent‚17.016687165019263 disk.dm-1.out‚46.53125 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.61328125 disk.drbd2265.in‚0 network.recv‚498525 disk.sda1.out‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2765.5703125 cpu‚25.314942656609066 disk.sda.out‚46.53125 disk.sda2.in‚0 disk.sda3.out‚46.53125" disk.dm-0.out‚4564.82763671875 disk.dm-1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚4564.82763671875 disk.drbd2080.in‚0 disk.dm-0.in‚0.09375ÍYÁô ‚(:… †­äÔÅÒŽ~ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä'jh9õ ‚(:® †­äÔÈú disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 2.30618e+06) disk./.usagePercent‚17.01877004231784 disk.dm-0.in‚0.09375 network.recv‚503269 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0$ memory_percent‚95.60368061065674 cpu‚25.311663973479384 disk.dm-1.out‚47.0625 disk.nvme0n1.in‚0.09375 disk.sda3.out‚47.0625 disk.drbd2080.in‚0( proc.memory.availableMB‚ 720.29296875 disk.rbd1.out‚ 0.61328125& disk.nvme0c0n1.out‚4615.1455078125$ disk.nvme0n1.out‚4615.1455078125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda.out‚47.0625 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2765.5703125% disk./.usageGB‚253.59197235107422! disk.dm-0.out‚4615.1455078125 disk.drbd2266.in‚0* proc.memory.percent‚16.879701614379883 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0.09375 disk.sda2.in‚0 disk.drbd2265.out‚0‰3eV2ö - _wandbruntime‚1171  _runtime‚1171k[å É÷ à ' lr-AdamW/pg2‚0.00016839112888820409 trainer/global_step‚1499 % _timestamp‚1.7884176717828007e+09 _runtime‚1171.520323493 _step‚119 ' lr-AdamW/pg1‚0.00016839112888820409®íÍø Ç ' lr-AdamW/pg1‚0.00016839112888820409 ' lr-AdamW/pg2‚0.00016839112888820409 trainer/global_step‚1499 % _timestamp‚1.7884176717828007e+09 _runtime‚1171.520323493 _step‚119wlŸèí2ù - _wandbruntime‚1171  _runtime‚1171¾ÕãÆ'ú ÀO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12667452 trainer/global_step‚1499 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1252999593984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85843790143951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96199048462848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19881682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788417671783461e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28257980 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252343232512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7214228 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88119762166272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1316115444736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1171.521207651 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88119087801856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12667454 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84595386076160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624ÞÄy-q û ÄO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88119762166272 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96198858800640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1316115444736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1171.521207651 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88119087801856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21712374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12667454 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86866762572288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88119087801856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84595386076160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86866744569344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19881227 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1315949722624 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12667452 trainer/global_step‚1499 NFDeviceStatsMoniaz’­Ytor.on_train_batch_start/inactive_split.small_pool.peak‚889 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28237944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1252999593984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49970356 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚85843790143951 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96199048462848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19881682 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189662208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚94882909078016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788417671783461e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28257980 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1252999593984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7213775 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21712376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28257980 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252343232512 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7214228 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86866762572288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0x€ã2ü - _wandbruntime‚1172  _runtime‚1172¿èMÑ&ý ËM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21726392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86923115611136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1253819035136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253162660352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1316820315648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12675622 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚1499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28256398 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884176724972684e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÊRÚtÕ&þ ÏM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21726390 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7218868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85898839232367 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚1499 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84650290206720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50002852 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚94944559815680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1248549025647 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86923115611136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28256398 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96261380131328 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884176724972684e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86923097608192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253162660352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28256398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50002852 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49982788 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21726392 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49982788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86923115611136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19894490 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96261561391616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1253819035136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚94944583755776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1249203463603 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86923097608192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19894051 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7218431 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1253162660352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28276460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1316820315648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21726390 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12675622 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88176934646272 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88176934646272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚85899510709683 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚422 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20518551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1288567930091 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90977725550080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13075852 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89684396511744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1359077341696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88629160534251 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20518551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51597208 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚420 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97965445961216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1288567930091 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚422 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90978399914496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29184032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚1549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1294148466176 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ! _timestamp‚1.788417709443e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51629704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13084022 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7447647 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89740767553536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22427192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88684881099775 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1294804840960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87395513774080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1289384365055 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1294804840960 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99386859384832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1290038803011 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51609640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20531226 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91034898016768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚1549 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51629704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22427190 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20531669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22427192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91035572394496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89740749550592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98027120638976 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1294148466176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91035572394496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1294148466176 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚99387061616640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1359940977664 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ! _timestamp‚1.788417709443e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0}MÍ2- _wandbruntime‚1209  _runtime‚1209ئtWÑ 1loss/SRqqt2/detection_loss‚2.0142241562234946 0loss/SRqqt4/detection_loss‚1.462585805320596 _step‚126 0loss/FRt4/assignment_loss‚3.2197806690969775 /loss/FBt1/detection_loss‚1.9007466102055945 1loss/SRqqt3/assignment_loss‚2.054897734418858 0loss/SRqqt1/detection_loss‚2.517118192381329 1loss/SRqqt3/detection_loss‚2.1844859109368433 epoch‚1 /loss/FRt1/assignment_loss‚4.438747161473984 /loss/FRt3/assignment_loss‚4.905929838909822 /loss/FRt2/assignment_loss‚3.777126507325606 2loss/SRqqt1/assignment_loss‚2.1826361475719347 1loss/FBt2/assignment_loss‚0.28890422979989694 /loss/FBt3/detection_loss‚2.9700571298599243 _runtime‚1209.182245967 2loss/SRqqt2/assignment_loss‚1.7845735480559284 .loss/FRt1/detection_loss‚2.409293899169335 .loss/FBt2/detection_loss‚6.323149959246318 /loss/FBt4/detection_loss‚1.1754262785642193 trainer/global_step‚1549 2loss/SRqqt4/assignment_loss‚1.6805825665682914 0loss/FBt1/assignment_loss‚0.4240861811692813 /loss/FBt3/assignment_loss‚0.543823798559606 .loss/FRt3/detection_loss‚2.778234040035921 /loss/FRt4/detection_loss‚2.0404235914109217 &loss/total_loss‚55.841928783519336 1loss/FBt4/assignment_loss‚0.28046190978881125 .loss/FRt2/detection_loss‚2.484632917425849 % _timestamp‚1.7884177094448156e+09öÄ$í[‘Õ % _timestamp‚1.7884177094448156e+09 1loss/SRqqt2/detection_loss‚2.0142241562234946 0loss/SRqqt4/detection_loss‚1.462585805320596 _step‚126 0loss/FRt4/assignment_loss‚3.2197806690969775 /loss/FBt1/detection_loss‚1.9007466102055945 1loss/SRqqt3/assignment_loss‚2.054897734418858 0loss/SRqqt1/detection_loss‚2.517118192381329 1loss/SRqqt3/detection_loss‚2.1844859109368433 epoch‚1 /loss/FRt1/assignment_loss‚4.438747161473984 /loss/FRt3/assignment_loss‚4.905929838909822 /loss/FRt2/assignment_loss‚3.777126507325606 2loss/SRqqt1/assignment_loss‚2.1826361475719347 1loss/FBt2/assignment_loss‚0.28890422979989694 /loss/FBt3/detection_loss‚2.9700571298599243 _runtime‚1209.182245967 2loss/SRqqt2/assignment_loss‚1.7845735480559284 .loss/FRt1/detection_loss‚2.409293899169335 .loss/FBt2/detection_loss‚6.323149959246318 /loss/FBt4/detection_loss‚1.1754262785642193 trainer/global_step‚1549 2loss/SRqqt4/assignment_loss‚1.6805825665682914 0loss/FBt1/assignment_loss‚0.4240861811692813 /loss/FBt3/assignment_loss‚0.543823798559606 .loss/FRt3/detection_loss‚2.778234040035921 /loss/FRt4/detection_loss‚2.0404235914109217 &loss/total_loss‚55.841928783519336 1loss/FBt4/assignment_loss‚0.28046190978881125 .loss/FRt2/detection_loss‚2.484632917425849~ô||6’‚(:’ ³­äÔÍÌ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wø\I“‚(:¾ ³­äÔù½€ disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd0.out‚0 disk.sda3.out‚ 48.83203125" disk.dm-0.out‚4776.10107421875 disk.dm-1.out‚ 48.83203125 disk.sda.in‚0 disk.sda.out‚ 48.83203125 disk.sda1.in‚0 disk.drbd1716.in‚0 network.recv‚519195% disk./.usageGB‚253.66867065429688 disk.nvme0n1.in‚4.09375 disk.nvme0c0n1.in‚4.09375* proc.memory.percent‚16.879701614379883 network.sent‚ 2.42614e+06 disk.rbd1.in‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚4776.10107421875 cpu‚25.301751642843143 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375 disk.dm-1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 719.6328125% disk.nvme0n1.out‚4776.10107421875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2765.5703125 disk.sda3.in‚0 disk.drbd2265.in‚0$ memory_percent‚95.60770988464355 proc.cpu.threads‚17* disk./.usagePercent‚17.023917329801275åË}ˆ’”‚(:‡ ­äÔ´Ýž~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•N=C•‚(:¸ ­äÔ‰™²€) disk./.usagePercent‚17.02464567123619 disk.drbd2080.in‚0! disk.dm-0.out‚4841.4033203125 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.09375 disk.sda2.out‚0" proc.memory.rssMB‚ 2765.5703125 disk.drbd2266.in‚0 network.sent‚ 2.534455e+06 network.recv‚526425$ disk.nvme0n1.out‚4841.4033203125' proc.memory.availableMB‚ 719.2890625 cpu‚25.298748955014688 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.09375& disk.nvme0c0n1.out‚4841.4033203125 disk.rbd0.out‚0% disk./.usageGB‚253.67952346801758 disk.drbd2265.out‚0 disk.dm-1.out‚ 49.2890625 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125* proc.memory.percent‚16.879701614379883 disk.rbd0.in‚0 disk.sda3.in‚0 disk.dm-0.in‚4.09375 disk.dm-1.in‚0 disk.sda3.out‚ 49.2890625$ memory_percent‚95.60980796813965 disk.sda.out‚ 49.2890625 disk.drbd1716.in‚0í…ÊÇ’–‚(:‡ Ñ­äÔÉù‹~ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´FF—‚(:» Ñ­äԠć disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.09375' disk.nvme0c0n1.out‚4871.23681640625* proc.memory.percent‚16.879701614379883 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda.out‚ 49.7890625 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 49.7890625 cpu‚25.295786465445435 network.recv‚530468 disk.dm-1.out‚ 49.7890625 disk.drbd2266.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2765.5703125" disk.dm-0.out‚4871.23681640625 disk.sda3.in‚0& proc.memory.availableMB‚ 719.515625 disk.dm-0.in‚4.09375 disk.drbd2266.in‚0% disk.nvme0n1.out‚4871.23681640625 disk.sda2.out‚0 network.sent‚ 2.540317e+06 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.rbd1.out‚ 0.61328125$ memory_percent‚95.60842514038086* disk./.usagePercent‚17.026000719114922% disk./.usageGB‚253.69971466064453 disk.drbd2080.out‚0î­ã2˜- _wandbruntime‚1245  _runtime‚1245‘ÏIÇ™Á _runtime‚1245.678223854 _step‚127 & lr-AdamW/pg1‚0.0001681524693338518 & lr-AdamW/pg2‚0.0001681524693338518 trainer/global_step‚1599 % _timestamp‚1.7884177459406643e+09_Ò˚Š& lr-AdamW/pg2‚0.0001681524693338518 trainer/global_step‚1599 % _timestamp‚1.7884177459406643e+09 _runtime‚1245.678223854 _step‚127 & lr-AdamW/pg1‚0.0001681524693338518šÚ©2›- _wandbruntime‚1245  _runtime‚1245Ó.b_Ê'œÄO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102449824094720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 1245.67920243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401841250304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1329403269499 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53224060 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 trainer/global_step‚1599””晤ÉO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93836363298304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101047982844416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚93837037662720 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1329403269499 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53224060 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 trainer/global_step‚1599 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21156154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13484252 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102449824094720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1330057707247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93836363298304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23113976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7671900 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚91415873878767 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7671449 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91415202401659 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1334314844160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚ 1245.67920243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23113974 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1334971205632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1401841250304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102512530442752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1330874142419 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚1246.378964558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93894210142720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788417746640924e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30108502 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23127992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30128564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402872920576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1599 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23127990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53236492 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚129 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088Íñf”Ù& ÓM R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91470922967183 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚1599 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93893535764992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21168370 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23127990 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93894210142720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7676390 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53236492 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1335134272000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚129 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53256556 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1402715600384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102512530442752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92558419495936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13492422 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23127992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1330874142419 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1330219704463 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚1246.378964558 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30128564 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93894210142720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788417746640924e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30108502 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23127992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23127990 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91471594444499 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30108502 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102512349182464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101109657522176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157320192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30128564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1402872920576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1335134272000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90140703262720 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384U‘#‰2¡- _wandbruntime‚1246  _runtime‚1246ñÔ!ÊU¢Ï /loss/FBt2/assignment_loss‚0.804015600681305 /loss/FBt2/detection_loss‚3.9718835592269897 /loss/FBt4/detection_loss‚1.3383786437846015 2loss/SRqqt2/assignment_loss‚1.7849565394222737 % _timestamp‚1.7884177466428566e+09 0loss/FBt1/assignment_loss‚0.6953325332549862 epoch‚1 -loss/FRt3/detection_loss‚2.72640542726259 1loss/SRqqt2/detection_loss‚2.2826909601688383 trainer/global_step‚1599 /loss/FRt1/assignment_loss‚4.329241690310565 2loss/SRqqt3/assignment_loss‚1.7465454700092475 _runtime‚1246.380275101 2loss/SRqqt1/assignment_loss‚2.2093282994525185 0loss/SRqqt1/detection_loss‚2.458921147831555 /loss/FBt1/detection_loss‚1.8051340897409953 .loss/FBt3/detection_loss‚3.035155153274536 %loss/total_loss‚54.42726396873739 0loss/FBt4/assignment_loss‚0.2590413356013442 0loss/FRt4/assignment_loss‚3.2187102688476443 1loss/SRqqt4/detection_loss‚1.3347774917282254 _step‚130 0loss/FBt3/assignment_loss‚0.5681782198759416 /loss/FRt4/detection_loss‚1.9274748205207288 1loss/SRqqt3/detection_loss‚2.4320122996966043 .loss/FRt1/detection_loss‚2.298117298971523 .loss/FRt2/detection_loss‚2.333586777959551 /loss/FRt2/assignment_loss‚4.398841763029293 /loss/FRt3/assignment_loss‚4.793561265275285 2loss/SRqqt4/assignment_loss‚1.6749733128102429Ù¾^ÿZ£Ô 0loss/FBt4/assignment_loss‚0.2590413356013442 0loss/FRt4/assignment_loss‚3.2187102688476443 1loss/SRqqt4/detection_loss‚1.3347774917282254 _step‚130 0loss/FBt3/assignment_loss‚0.5681782198759416 /loss/FRt4/detection_loss‚1.9274748205207288 1loss/SRqqt3/detection_loss‚2.4320122996966043 .loss/FRt1/detection_loss‚2.298117298971523 .loss/FRt2/detection_loss‚2.333586777959551 /loss/FRt2/assignment_loss‚4.398841763029293 /loss/FRt3/assignment_loss‚4.793561265275285 2loss/SRqqt4/assignment_loss‚1.6749733128102429 /loss/FBt2/assignment_loss‚0.804015600681305 /loss/FBt2/detection_loss‚3.9718835592269897 /loss/FBt4/detection_loss‚1.3383786437846015 2loss/SRqqt2/assignment_loss‚1.7849565394222737 % _timestamp‚1.7884177466428566e+09 0loss/FBt1/assignment_loss‚0.6953325332549862 epoch‚1 -loss/FRt3/detection_loss‚2.72640542726259 1loss/SRqqt2/detection_loss‚2.2826909601688383 trainer/global_step‚1599 /loss/FRt1/assignment_loss‚4.329241690310565 2loss/SRqqt3/assignment_loss‚1.7465454700092475 _runtime‚1246.380275101 2loss/SRqqt1/assignment_loss‚2.2093282994525185 0loss/SRqqt1/detection_loss‚2.458921147831555 /loss/FBt1/detection_loss‚1.8051340897409953 .loss/FBt3/detection_loss‚3.035155153274536 %loss/total_loss‚54.42726396873739‚3 fõœ¤‚(:‘ à­äÔ•ï~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.372' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.27440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RH™ÁB¥‚(:· à­äÔêƨ€ disk.dm-0.out‚ 4947.09765625 disk.drbd2266.in‚0* disk./.usagePercent‚17.026048336516293( proc.memory.availableMB‚ 719.46484375" proc.memory.rssMB‚ 2765.5703125* proc.memory.percent‚16.879701614379883 proc.cpu.threads‚17" disk.nvme0n1.out‚ 4947.09765625$ disk.nvme0c0n1.out‚ 4947.09765625 disk.sda2.out‚0 disk.dm-1.out‚ 50.44921875 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.sda.in‚0 disk.sda.out‚ 50.44921875 network.sent‚ 2.546633e+06% disk./.usageGB‚253.70042419433594 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.09375 disk.rbd1.in‚0$ memory_percent‚95.60873508453369 disk.nvme0c0n1.in‚4.09375 cpu‚25.292755482489593 network.recv‚535612 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.61328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 50.44921875 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375H ?9’¦‚(:‡ ï­äÔÄð¡~ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š‚D :§‚(:¯ ï­äÔ÷¸Ë disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 4993.55859375 disk.nvme0n1.in‚4.09375* proc.memory.percent‚16.879701614379883 network.recv‚542097 disk.drbd2265.in‚0 disk.dm-0.in‚4.09375 disk.sda.out‚ 50.921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.61328125 cpu‚25.289906011551583 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.sda.in‚0 disk.drbd1716.out‚0$ disk.nvme0c0n1.out‚ 4993.55859375" proc.memory.rssMB‚ 2765.5703125* disk./.usagePercent‚17.028290962516238 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚95.6111192703247( proc.memory.availableMB‚ 719.07421875$ disk./.usageGB‚253.7338409423828 disk.sda3.out‚ 50.921875 disk.drbd2080.in‚0 disk.dm-1.out‚ 50.921875" disk.nvme0n1.out‚ 4993.55859375 disk.nvme0c0n1.in‚4.09375 network.sent‚ 2.654458e+06 disk.drbd2266.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2266.out‚0Ù¾¢u2¨- _wandbruntime‚1282  _runtime‚1282v¥ºÉ©Ã ' lr-AdamW/pg1‚0.00016800692765440536 ' lr-AdamW/pg2‚0.00016800692765440536 trainer/global_step‚1649 % _timestamp‚1.7884177824476056e+09 _runtime‚1282.185142069 _step‚131ÖO{ÎªÈ _step‚131 ' lr-AdamW/pg1‚0.00016800692765440536 ' lr-AdamW/pg2‚0.00016800692765440536 trainer/global_step‚1649 % _timestamp‚1.7884177824476056e+09 _runtime‚1282.185142069ƒ­É;?2«- _wandbruntime‚1282  _runtime‚1282+oZý ¬ÃO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370238608907 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 trainer/global_step‚1649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23814774 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104130519727616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31036136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23814776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonit<âÚ*Éor.on_train_batch_start/allocation.large_pool.allocated‚23814776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92831005660160 _step‚132 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96695675410944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96695001046528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788417782448327e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13892654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7898480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105575662560768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399æŠÎ'­ÈO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96695675410944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54850912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96695001046528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95319700396544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370893046655 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95319718399488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1375300649984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788417782448327e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚92831022699520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96695001046528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13892654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7898480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1375300649984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7898017 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105575662560768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21791134 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370238608907 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1375957011456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94201915746175 trainer/global_step‚1649 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23814774 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104130519727616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13892652 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31016100 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94201244269067 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105575462412800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31036136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23814776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23814776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31016100 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95319700396544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚92831005660160 _step‚132„1Pà2®- _wandbruntime‚1282  _runtime‚1282·är0Ò&¯ÌM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7902481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 " _timestamp‚1.7884177831471e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95376053435392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96752847890944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95376053435392 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96752847890944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54883408 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚104192170465280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23828790 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13900820 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1376120077824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95376071438336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31034554 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94257636311907 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23828792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21803301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7902481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54863344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31054616…© ¢Ã2±- _wandbruntime‚1282  _runtime‚1282ÜO²É .loss/FRt2/detection_loss‚2.596461127201716 .loss/FBt3/detection_loss‚3.357225434647666 .loss/FBt2/detection_loss‚3.889138800757272 0loss/FBt3/assignment_loss‚0.4781316715311732 2loss/SRqqt4/assignment_loss‚1.7058534638615706 .loss/FRt4/detection_loss‚1.956890022707364 1loss/SRqqt3/detection_loss‚1.9943944088956143 1loss/FBt4/assignment_loss‚0.22100584532840217 .loss/FRt1/detection_loss‚2.474906879086648 0loss/SRqqt1/detection_loss‚2.658504989412096 epoch‚1 trainer/global_step‚1649 /loss/FRt1/assignment_loss‚4.850586075936595 /loss/FRt4/assignment_loss‚2.805987158465007 1loss/SRqqt3/assignment_loss‚2.271531845343874 /loss/FRt3/detection_loss‚2.6875201072011676 _runtime‚1282.886217055 /loss/FRt2/assignment_loss‚4.652746539645725 1loss/SRqqt1/assignment_loss‚2.253289375592161 0loss/SRqqt2/detection_loss‚2.418449155241251 $ _timestamp‚1.788417783148794e+09 _step‚134 /loss/FBt4/detection_loss‚1.5931556617320215 0loss/FBt1/assignment_loss‚0.6002258854465682 0loss/FBt2/assignment_loss‚0.9429565263645989 1loss/SRqqt4/detection_loss‚1.2374669866215797 .loss/FBt1/detection_loss‚1.984360040987239 $loss/total_loss‚56.6799463076821 /loss/FRt3/assignment_loss‚4.890400218963623 1loss/SRqqt2/assignment_loss‚2.158758086711168ŒÂªûT³Î 1loss/SRqqt3/assignment_loss‚2.271531845343874 /loss/FRt3/detection_loss‚2.6875201072011676 _runtime‚1282.886217055 /loss/FRt2/assignment_loss‚4.652746539645725 1loss/SRqqt1/assignment_loss‚2.253289375592161 0loss/SRqqt2/detection_loss‚2.418449155241251 $ _timestamp‚1.788417783148794e+09 _step‚134 /loss/FBt4/detection_loss‚1.5931556617320215 0loss/FBt1/assignment_loss‚0.6002258854465682 0loss/FBt2/assignment_loss‚0.9429565263645989 1loss/SRqqt4/detection_loss‚1.2374669866215797 .loss/FBt1/detection_loss‚1.984360040987239 $loss/total_loss‚56.6799463076821 /loss/FRt3/assignment_loss‚4.890400218963623 1loss/SRqqt2/assignment_loss‚2.158758086711168 .loss/FRt2/detection_loss‚2.596461127201716 .loss/FBt3/detection_loss‚3.357225434647666 .loss/FBt2/detection_loss‚3.889138800757272 0loss/FBt3/assignment_loss‚0.4781316715311732 2loss/SRqqt4/assignment_loss‚1.7058534638615706 .loss/FRt4/detection_loss‚1.956890022707364 1loss/SRqqt3/detection_loss‚1.9943944088956143 1loss/FBt4/assignment_loss‚0.22100584532840217 .loss/FRt1/detection_loss‚2.474906879086648 0loss/SRqqt1/detection_loss‚2.658504989412096 epoch‚1 trainer/global_step‚1649 /loss/FRt1/assignment_loss‚4.850586075936595 /loss/FRt4/assignment_loss‚2.805987158465007†ÉkÑd´‚(:’ þ­äÔøβ~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÍ@µ‚(:µ þ­äÔ‚½· disk.drbd2265.in‚0% disk./.usageGB‚253.75127029418945 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.09375 disk.nvme0c0n1.in‚4.09375 disk.sda3.in‚0 network.recv‚546654 disk.dm-1.in‚0 disk.dm-1.out‚ 51.453125 disk.rbd1.out‚ 0.61328125 disk.sda.out‚ 51.453125 disk.drbd2080.out‚0" disk.dm-0.out‚5043.77587890625* proc.memory.percent‚16.879701614379883 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0% disk.nvme0n1.out‚5043.77587890625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5043.77587890625 disk.drbd2265.out‚0 disk.dm-0.in‚4.09375 cpu‚25.28736767706712 disk.sda2.out‚0) disk./.usagePercent‚17.02946066094018 disk.rbd1.in‚0 disk.sda.in‚0% proc.memory.availableMB‚ 719.03125" proc.memory.rssMB‚ 2765.5703125 network.sent‚ 2.660504e+06 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 51.453125 proc.cpu.threads‚17$ memory_percent‚95.61138153076172&8DF¶‚(:’ ®äÔ³¡«~ gpu.0.gpu‚53.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úýà2J·‚(:¿ ®äÔî¨Á€ disk.dm-0.in‚4.09375 disk.sda.in‚0 disk.drbd1716.in‚0$ memory_percent‚95.61147689819336* proc.memory.percent‚16.879725456237793 network.sent‚ 2.768721e+06 network.recv‚553743 disk.drbd2265.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚5091.39697265625 proc.cpu.threads‚17 disk.drbd2265.out‚0" disk.dm-0.out‚5091.39697265625 disk.sda1.in‚0 disk.rbd1.out‚ 0.61328125 disk.sda.out‚ 51.98046875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2765.57421875 cpu‚25.284541580047748 disk.sda3.out‚ 51.98046875 disk.drbd2080.in‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 719.015625 disk.nvme0c0n1.in‚4.09375 disk.sda3.in‚0 disk.dm-1.out‚ 51.98046875 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.09375% disk./.usageGB‚253.77197647094727' disk.nvme0c0n1.out‚5091.39697265625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0* disk./.usagePercent‚17.030850269836037 disk.drbd2080.out‚0½52¸- _wandbruntime‚1319  _runtime‚1319Â!76ɹà ' lr-AdamW/pg1‚0.00016784395060106034 ' lr-AdamW/pg2‚0.00016784395060106034 trainer/global_step‚1699 % _timestamp‚1.7884178194209363e+09 _runtime‚1319.158460154 _step‚135íb¥ ÎºÈ ' lr-AdamW/pg1‚0.00016784395060106034 ' lr-AdamW/pg2‚0.00016784395060106034 trainer/global_step‚1699 % _timestamp‚1.7884178194209363e+09 _runtime‚1319.158460154 _step‚135‡vh$2»- _wandbruntime‚1319  _runtime‚1319•’Ÿ^Ê'¼ÄO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22426995 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24515574 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22426539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8125487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚1699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56457726 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24515576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1319.159589555 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99554313159168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872ÖîgËÏ'½ÉO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98137352338944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1411073948315 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99554313159168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107213056610816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24515574 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22426995 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24515574 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22426539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95576212188160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56477764 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1487939000320 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95576229227520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8125487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚1699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1416286455808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31942152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98137370341888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31962188 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31942152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14301054 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56457726 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1416942817280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99553638794752 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8125941 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1416286455808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108700995611136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31962188 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24515576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1319.159589555 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24529590 MCDeviceSt‘u’÷8!atsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31960606 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8130009 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22439674 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31960606 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99611485639168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1417762258432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 _runtime‚ 1319.85723275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107274707348480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1417105883648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108763501957632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8130452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95631133358080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411890383279 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884178201190014e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1412544821235 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚98193723380736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8130452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191746048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108763693703680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95631133358080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1411890383279 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884178201190014e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56490196 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1417105883648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56510260 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98193723380736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24529590 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31960606 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8130009 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24529590 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22439674 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31980668 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99610811261440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31960606 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99611485639168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98193705377792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56490196 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98193705377792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24529592 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1417762258432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22439229 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14309220 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167805952 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97043678179315 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚99611485639168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1417762258432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95631116318720 _runtime‚ 1319.85723275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107274707348480 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24529592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1417105883648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1488794609152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728‰þDßf2Á- _wandbruntime‚1319  _runtime‚13195-ÃUÂÏ .loss/FBt3/detection_loss‚3.906645618379116 /loss/FRt2/assignment_loss‚4.270757743290493 2loss/SRqqt4/assignment_loss‚1.4896366296646495 epoch‚1 /loss/FRt1/detection_loss‚2.6999339595917733 .loss/FRt2/detection_loss‚2.954131187711443 /loss/FRt3/detection_loss‚2.8678946178406477 .loss/FRt4/detection_loss‚1.771812967577223 1loss/SRqqt1/detection_loss‚2.5617564049634067 0loss/SRqqt2/detection_loss‚2.395615551117304 1loss/FBt3/assignment_loss‚0.38146309950752766 0loss/FBt2/assignment_loss‚0.7168107479034612 /loss/FBt4/detection_loss‚0.9256207928017436 _step‚138 2loss/SRqqt1/assignment_loss‚2.7697087508259397 /loss/FBt2/detection_loss‚3.5653980374336243 _runtime‚ 1319.85827711 0loss/FRt1/assignment_loss‚5.2149595099110755 /loss/FRt4/assignment_loss‚2.751075169797671 # _timestamp‚1.78841782012083e+09 0loss/FRt3/assignment_loss‚4.5396560952067375 %loss/total_loss‚59.10353986771861 trainer/global_step‚1699 1loss/FBt4/assignment_loss‚0.32277707141986717 1loss/SRqqt2/assignment_loss‚2.580517546550648 2loss/SRqqt3/assignment_loss‚1.7242291937100476 /loss/FBt1/assignment_loss‚0.703789485352378 0loss/SRqqt3/detection_loss‚2.764290821963343 1loss/SRqqt4/detection_loss‚1.2035547009242389 /loss/FBt1/detection_loss‚4.0215041642742495›ÖâµZÃÔ 2loss/SRqqt1/assignment_loss‚2.7697087508259397 /loss/FBt2/detection_loss‚3.5653980374336243 _runtime‚ 1319.85827711 0loss/FRt1/assignment_loss‚5.2149595099110755 /loss/FRt4/assignment_loss‚2.751075169797671 # _timestamp‚1.78841782012083e+09 0loss/FRt3/assignment_loss‚4.5396560952067375 %loss/total_loss‚59.10353986771861 trainer/global_step‚1699 1loss/FBt4/assignment_loss‚0.32277707141986717 1loss/SRqqt2/assignment_loss‚2.580517546550648 2loss/SRqqt3/assignment_loss‚1.7242291937100476 /loss/FBt1/assignment_loss‚0.703789485352378 0loss/SRqqt3/detection_loss‚2.764290821963343 1loss/SRqqt4/detection_loss‚1.2035547009242389 /loss/FBt1/detection_loss‚4.0215041642742495 .loss/FBt3/detection_loss‚3.906645618379116 /loss/FRt2/assignment_loss‚4.270757743290493 2loss/SRqqt4/assignment_loss‚1.4896366296646495 epoch‚1 /loss/FRt1/detection_loss‚2.6999339595917733 .loss/FRt2/detection_loss‚2.954131187711443 /loss/FRt3/detection_loss‚2.8678946178406477 .loss/FRt4/detection_loss‚1.771812967577223 1loss/SRqqt1/detection_loss‚2.5617564049634067 0loss/SRqqt2/detection_loss‚2.395615551117304 1loss/FBt3/assignment_loss‚0.38146309950752766 0loss/FBt2/assignment_loss‚0.7168107479034612 /loss/FBt4/detection_loss‚0.9256207928017436 _step‚138Š±m`U’Ä‚(:‡ œ®äÔóâ¬~ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï@#?FÅ‚(:» œ®äÔÚª¹€ disk.drbd2265.out‚0 disk.sda.out‚ 52.6015625# proc.memory.rssMB‚ 2765.57421875* proc.memory.percent‚16.879725456237793% disk./.usageGB‚253.78613662719727 disk.sda3.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚5163.84521484375 disk.rbd0.out‚0 network.recv‚558915 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.09375 disk.dm-1.out‚ 52.6015625) disk./.usagePercent‚17.03180056980314 disk.nvme0c0n1.in‚4.09375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.09375( proc.memory.availableMB‚ 718.76953125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚5163.84521484375 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.sent‚ 2.775088e+06 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0 disk.drbd1716.in‚0 cpu‚25.28211587669253 disk.sda3.out‚ 52.6015625" disk.dm-0.out‚5163.84521484375 disk.drbd2266.out‚0# memory_percent‚95.6129789352417 disk.sda1.out‚0 disk.sda2.out‚0æ"’Æ‚(:‡ «®äÔå‘™~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.dTIÇ‚(:¾ «®äÔöèŠ) disk./.usagePercent‚17.03294671553393% disk./.usageGB‚253.80321502685547 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0% disk.nvme0n1.out‚5210.10693359375# proc.memory.rssMB‚ 2765.57421875 network.recv‚565508 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.sda.in‚0 network.sent‚ 2.882698e+06" disk.dm-0.out‚5210.10693359375 cpu‚25.27935033291664 disk.sda2.in‚0 disk.dm-0.in‚4.09375 disk.dm-1.out‚ 53.12109375 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.09375$ memory_percent‚95.61171531677246 proc.cpu.threads‚17 disk.rbd1.out‚ 0.61328125 disk.drbd1716.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚5210.10693359375 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 53.12109375' proc.memory.availableMB‚ 718.9765625* proc.memory.percent‚16.879725456237793 disk.nvme0c0n1.in‚4.09375 disk.sda2.out‚0 disk.sda3.out‚ 53.12109375 disk.drbd1716.out‚0°„¦È‚(:’ º®äÔ–ºÿ} gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.339' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.867799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×Пù:É‚(:¯ º®äÔ¥¡‚€ disk.nvme0n1.out‚ 5262.328125 disk.sda2.in‚0' proc.memory.availableMB‚ 718.7109375* proc.memory.percent‚16.879725456237793 cpu‚25.27689068468415 disk.dm-0.in‚4.09375 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 5262.328125 disk.drbd2265.out‚0$ memory_percent‚95.61333656311035# proc.memory.rssMB‚ 2765.57421875 disk.dm-1.out‚ 53.45703125 disk.sda.out‚ 53.45703125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 5262.328125 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 0.61328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.09375 network.sent‚ 2.888676e+06) disk./.usagePercent‚17.03453985041951% disk./.usageGB‚253.82695388793945 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 53.45703125 network.recv‚569872 disk.sda.in‚0 disk.nvme0n1.in‚4.09375ëî´!2Ê- _wandbruntime‚1355  _runtime‚1355‰©ÜÏÇËÁ trainer/global_step‚1749 $ _timestamp‚1.788417856106934e+09 _runtime‚ 1355.84444724 _step‚139 ' lr-AdamW/pg1‚0.00016766357213799036 ' lr-AdamW/pg2‚0.00016766357213799036ŸåùÌÌÆ _step‚139 ' lr-AdamW/pg1‚0.00016766357213799036 ' lr-AdamW/pg2‚0.00016766357213799036 trainer/global_step‚1749 $ _timestamp‚1.788417856106934e+09 _runtime‚ 1355.84444724‹/Þ;2Í- _wandbruntime‚1355  _runtime‚1355–°È×ÎÍO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58104616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Šõó¥¶JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110295617434112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99773328003883 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25216376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554 trainer/global_step‚1749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944ÊiœØ'ÏÒO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98321435755520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14709454 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1531079217152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25216374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58104616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚102412950907392 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1457272261632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102412276542976 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58084578 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1457928623104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110295617434112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99773328003883 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25216376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1531238647808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353554 trainer/global_step‚1749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23062576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14709452 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111889368474112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 1356.53502704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23075650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚141 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110357268171776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25230392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0xGaPA"ÒÛM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99829720046723 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32886658 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14717622 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884178567970066e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25230390 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1458748064256 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1458091689472 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58117048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23075650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1453380160643 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚141 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1531922010624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102470123387392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚101011375323136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚102469449009664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110357244231680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58137112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25230390 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58137112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101011375323136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32906720 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1458091689472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14717620 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110357268171776 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1532100302336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8357581 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101011357320192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25230392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32906720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚99829048569407 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8358028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25230392 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102470123387392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚111889166242304 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1452725722687 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32886658 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚111889368474112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batcì"Í h_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23075201 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178291712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚ 1356.53502704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101011357320192 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101066642283520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105271588655616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884178930942156e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59711430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1498258067456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1392.832038799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8582033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105270914291200 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23699467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚101066642283520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105271588655616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103772656223744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25917174 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884178930942156e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23699887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15117852 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105270914291200 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59711430 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25917174 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚101066625244160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33794256 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1498914428928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105271588655616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814292 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8581615 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103772674226688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113378130377216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1498258067456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114952387907584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25917176 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103772674226688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102559369871291 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚1392.832038799 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185467904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1799 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33794256 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8582033 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15117854 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15126020 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_p ‚Ý_ool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1799 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33812710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101121529374720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1494215500051 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33812710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚145 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1493561062095 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102615090436815 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102615761914131 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113439805054976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430‰ ©æ&âàM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25931190 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103829027265536 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25931192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33812710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚101121529374720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1494215500051 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33812710 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103829027265536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25931190 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59743900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105328086757888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚145 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1493561062095 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1574921943040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102615090436815 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172000256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102615761914131 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚101121546414080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105328761135616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115014703057920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113439805054976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23712111 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103829009262592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105328761135616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15126020 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33832772 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1799 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59763964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59763964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1‘|hû2ã- _wandbruntime‚1393  _runtime‚1393-]ûASäÍ /loss/FRt1/detection_loss‚2.2444131904178195 0loss/SRqqt3/detection_loss‚2.143128331218447 1loss/SRqqt4/detection_loss‚1.3033904795041855 0loss/FBt1/assignment_loss‚0.6061336035529773 0loss/FBt2/assignment_loss‚0.5965920062735677 _step‚146 1loss/SRqqt1/detection_loss‚2.1435957245710417 1loss/SRqqt2/detection_loss‚2.3844772818926216 %loss/total_loss‚56.44859548448076 0loss/FBt4/assignment_loss‚0.2139554948919193 .loss/FBt1/detection_loss‚2.267689296975732 _runtime‚1393.520051775 .loss/FRt3/assignment_loss‚4.04667321411339 0loss/SRqqt2/assignment_loss‚2.31859539650582 % _timestamp‚1.7884178937826028e+09 /loss/FRt2/detection_loss‚2.7171242253200427 0loss/FRt4/assignment_loss‚2.0449577481188674 .loss/FRt3/detection_loss‚2.741001353070543 trainer/global_step‚1799 0loss/FBt3/assignment_loss‚0.4870994538067066 /loss/FBt2/detection_loss‚4.7620717734098434 /loss/FBt4/detection_loss‚1.1987613403809168 epoch‚1 /loss/FRt2/assignment_loss‚5.349320163597932 1loss/SRqqt1/assignment_loss‚2.166070910488687 1loss/SRqqt3/assignment_loss‚2.100404636136123 2loss/SRqqt4/assignment_loss‚1.3650970555403654 /loss/FRt4/detection_loss‚2.2178369456149163 .loss/FBt3/detection_loss‚5.047311723232269 /loss/FRt1/assignment_loss‚3.982894135846032K„™åXåÒ 0loss/FBt4/assignment_loss‚0.2139554948919193 .loss/FBt1/detection_loss‚2.267689296975732 _runtime‚1393.520051775 .loss/FRt3/assignment_loss‚4.04667321411339 0loss/SRqqt2/assignment_loss‚2.31859539650582 % _timestamp‚1.7884178937826028e+09 /loss/FRt2/detection_loss‚2.7171242253200427 0loss/FRt4/assignment_loss‚2.0449577481188674 .loss/FRt3/detection_loss‚2.741001353070543 trainer/global_step‚1799 0loss/FBt3/assignment_loss‚0.4870994538067066 /loss/FBt2/detection_loss‚4.7620717734098434 /loss/FBt4/detection_loss‚1.1987613403809168 epoch‚1 /loss/FRt2/assignment_loss‚5.349320163597932 1loss/SRqqt1/assignment_loss‚2.166070910488687 1loss/SRqqt3/assignment_loss‚2.100404636136123 2loss/SRqqt4/assignment_loss‚1.3650970555403654 /loss/FRt4/detection_loss‚2.2178369456149163 .loss/FBt3/detection_loss‚5.047311723232269 /loss/FRt1/assignment_loss‚3.982894135846032 /loss/FRt1/detection_loss‚2.2444131904178195 0loss/SRqqt3/detection_loss‚2.143128331218447 1loss/SRqqt4/detection_loss‚1.3033904795041855 0loss/FBt1/assignment_loss‚0.6061336035529773 0loss/FBt2/assignment_loss‚0.5965920062735677 _step‚146 1loss/SRqqt1/detection_loss‚2.1435957245710417 1loss/SRqqt2/detection_loss‚2.3844772818926216 %loss/total_loss‚56.44859548448076’h„ÏÌæ‚(:’ ç®äÔœ•Œ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.937' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.187400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú&<:ç‚(:¯ ç®äÔàƒš€ disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚5423.5068359375 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚55.4375 network.sent‚ 3.008059e+06$ disk./.usageGB‚253.8525161743164! disk.dm-0.out‚5423.5068359375 disk.nvme0n1.in‚4.09375$ memory_percent‚95.61247825622559* proc.memory.percent‚16.879725456237793 network.recv‚585034 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚5423.5068359375 disk.sda3.out‚55.4375 disk.drbd2080.in‚0 cpu‚25.269523472829533 disk.dm-0.in‚4.09375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.09375 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2765.57421875) disk./.usagePercent‚17.03625535690645 disk.dm-1.out‚55.4375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 0.61328125 disk.sda.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 718.8515625 proc.cpu.threads‚17fã²ÿ’è‚(:‡ ö®äÔöÇŒ~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0啽2é‚(:§ ö®äÔ¾•€) disk./.usagePercent‚17.03716981581876% disk./.usageGB‚253.86614227294922 disk.sda3.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚56.1875 disk.sda.out‚56.1875 cpu‚25.26723201207246 disk.nvme0n1.in‚4.09375 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375* proc.memory.percent‚16.879725456237793 network.sent‚ 3.116449e+06 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda.in‚0$ memory_percent‚95.61245441436768# proc.memory.rssMB‚ 2765.57421875 proc.cpu.threads‚17! disk.nvme0n1.out‚ 5478.3984375# disk.nvme0c0n1.out‚ 5478.3984375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 5478.3984375 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.09375 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 718.85546875 network.recv‚592195 disk.drbd1716.out‚0 disk.sda3.out‚56.1875 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.69921875vå¢"’ê‚(:‡ …¯äÔþÈ…~ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 :#š>ë‚(:³ …¯äÔÙ–Š€ network.recv‚596403* disk./.usagePercent‚17.037044116119446 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.09375# memory_percent‚95.6125020980835 disk.sda3.in‚0# disk.nvme0n1.out‚5527.705078125* proc.memory.percent‚16.879725456237793% disk.nvme0c0n1.out‚5527.705078125 disk.sda2.out‚0 disk.sda3.out‚ 56.796875 disk.drbd2080.in‚0 disk.sda.out‚ 56.796875 network.sent‚ 3.122336e+06$ disk./.usageGB‚253.8642692565918 disk.drbd2265.out‚0 cpu‚25.264989579864896 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.09375 disk.dm-0.in‚4.09375 disk.dm-0.out‚5527.705078125 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.76171875 disk.drbd1716.out‚0 disk.dm-1.out‚ 56.796875# proc.memory.rssMB‚ 2765.57421875 disk.drbd1716.in‚0( proc.memory.availableMB‚ 718.84765625"÷8Â2ì- _wandbruntime‚1429  _runtime‚1429BTÆíÀ _step‚147 & lr-AdamW/pg1‚0.0001672507649636748 & lr-AdamW/pg2‚0.0001672507649636748 trainer/global_step‚1849 % _timestamp‚1.7884179296520197e+09 _runtime‚ 1429.3897373363 ËîÅ & lr-AdamW/pg1‚0.0001672507649636748 & lr-AdamW/pg2‚0.0001672507649636748 trainer/global_step‚1849 % _timestamp‚1.7884179296520197e+09 _runtime‚ 1429.38973733 _step‚147“V€¡ˆ2ï- _wandbruntime‚1429  _runtime‚1429ÿñŽ&ðÑO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103811848811520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103811831772160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118078042485248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26617974 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8810037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106590326169088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚1849 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freô&ÙŒÉed‚34720308 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1617351284736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752Òý&Ü'ñÖO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34740344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106590326169088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24336753 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24336289 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34720308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15526252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 trainer/global_step‚1849 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116460667260416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61358320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8810499 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118077846531584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34720308 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1617351284736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108130226403840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1539900234752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116460691200512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103811848811520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚106590308166144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚103811831772160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118078042485248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26617976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚106590326169088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1534234404287 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61338282 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1539243873280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108130226403840 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚148 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26617974 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108129552039424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8810037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26631990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106646679207936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34738762 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1430.075563209 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34758824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26631992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 _step‚149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106646661204992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103866735902720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118140362069504úñOLæ&ôàM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15534422 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108187398883840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61370752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1618044071424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1430.075563209 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24349019 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34758824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61390816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1535050839459 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26631992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8815047 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1540063301120 _step‚149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106646661204992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚103866735902720 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118140362069504 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1540063301120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1534396401503 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34738762 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1540719675904 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105401803781539 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26631990 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1618207683072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8814599 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚106646679207936 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61390816 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103866752942080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26631990 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34738762 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚1849 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728•ÝŽ™s2õ- _wandbruntime‚1430  _runtime‚1430]L±\[öÕ /loss/FRt3/assignment_loss‚4.482497450632927 2loss/SRqqt3/assignment_loss‚2.3267152549491987 2loss/SRqqt4/assignment_loss‚1.6102706489781307 2loss/SRqqt1/assignment_loss‚1.9780332224709647 0loss/FBt1/assignment_loss‚0.6696105429104386 /loss/FRt2/detection_loss‚2.8486786475687316 1loss/SRqqt1/detection_loss‚2.3282249910490855 1loss/SRqqt3/detection_loss‚2.4305848537219896 1loss/SRqqt4/detection_loss‚1.1489367602977816 /loss/FBt1/detection_loss‚2.5648616467203413 /loss/FBt2/detection_loss‚3.1005284828799113 /loss/FRt2/assignment_loss‚4.147483731761123 1loss/FBt2/assignment_loss‚0.31711257264708237 1loss/SRqqt2/detection_loss‚2.2372371452775868 /loss/FBt4/detection_loss‚0.9077976720135023 %loss/total_loss‚52.73530743781305 % _timestamp‚1.7884179303392398e+09 0loss/FRt4/assignment_loss‚2.5495811934564627 1loss/FBt4/assignment_loss‚0.34412315781561875 .loss/FRt1/detection_loss‚2.426828032030779 /loss/FBt3/detection_loss‚2.5769097154790703 epoch‚1 .loss/FRt1/assignment_loss‚4.33514940738678 /loss/FRt3/detection_loss‚2.6211210000209317 1loss/SRqqt2/assignment_loss‚1.818323952569203 0loss/FBt3/assignment_loss‚0.7436780963431705 /loss/FRt4/detection_loss‚2.2210192588322304 trainer/global_step‚1849 _runtime‚1430.076657674 _step‚150aK_÷Ú 0loss/FRt4/assignment_loss‚2.5495811934564627 1loss/FBt4/assignment_loss‚0.34412315781561875 .loss/FRt1/detection_loss‚2.426828032030779 /loss/FBt3/detection_loss‚2.5769097154790703 epoch‚1 .loss/FRt1/assignment_loss‚4.33514940738678 /loss/FRt3/detection_loss‚2.6211210000209317 1loss/SRqqt2/assignment_loss‚1.818323952569203 0loss/FBt3/assignment_loss‚0.7436780963431705 /loss/FRt4/detection_loss‚2.2210192588322304 trainer/global_step‚1849 _runtime‚1430.076657674 _step‚150 /loss/FRt3/assignment_loss‚4.482497450632927 2loss/SRqqt3/assignment_loss‚2.3267152549491987 2loss/SRqqt4/assignment_los@wGÕÃs‚1.6102706489781307 2loss/SRqqt1/assignment_loss‚1.9780332224709647 0loss/FBt1/assignment_loss‚0.6696105429104386 /loss/FRt2/detection_loss‚2.8486786475687316 1loss/SRqqt1/detection_loss‚2.3282249910490855 1loss/SRqqt3/detection_loss‚2.4305848537219896 1loss/SRqqt4/detection_loss‚1.1489367602977816 /loss/FBt1/detection_loss‚2.5648616467203413 /loss/FBt2/detection_loss‚3.1005284828799113 /loss/FRt2/assignment_loss‚4.147483731761123 1loss/FBt2/assignment_loss‚0.31711257264708237 1loss/SRqqt2/detection_loss‚2.2372371452775868 /loss/FBt4/detection_loss‚0.9077976720135023 %loss/total_loss‚52.73530743781305 % _timestamp‚1.7884179303392398e+09–¼ÿ)]’ø‚(:‡ ”¯äÔÜ ˜~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËÐ?Hù‚(:½ ”¯äÔ³»™€ disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.rbd0.out‚0 network.sent‚ 3.128641e+06% disk./.usageGB‚253.88980102539062 disk.sda3.out‚ 57.44921875 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚5593.4716796875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.09375' proc.memory.availableMB‚ 718.7890625 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-1.out‚ 57.44921875 disk.drbd2266.out‚0* proc.memory.percent‚16.879725456237793 network.recv‚601253 disk.drbd2265.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚5593.4716796875# proc.memory.rssMB‚ 2765.57421875! disk.dm-0.out‚5593.4716796875 disk.drbd2080.in‚0 disk.rbd1.out‚ 0.76171875 disk.sda.out‚ 57.44921875 disk.sda3.in‚0 disk.dm-0.in‚4.09375 cpu‚25.262785694454003* disk./.usagePercent‚17.038757574546118 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0$ memory_percent‚95.61285972595215ï Âzœú‚(:‘ £¯äÔ¸ÊØ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.382000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×­¹iGû‚(:¼ £¯äÔèµ½ network.recv‚607926$ disk.nvme0n1.out‚5628.5107421875 disk.sda.out‚ 57.92578125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 network.sent‚ 3.236467e+06 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 cpu‚25.26058553024401 disk.sda1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚5628.5107421875 disk.sda3.in‚0 disk.sda3.out‚ 57.92578125 proc.cpu.threads‚17* disk./.usagePercent‚17.036334719242067% disk./.usageGB‚253.85369873046875 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.09375$ memory_percent‚95.61142921447754! disk.dm-0.out‚5628.5107421875 disk.rbd1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 719.0234375* proc.memory.percent‚16.879749298095703 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.7939453125 disk.sda.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.09375! proc.memory.rssMB‚ 2765.578125 disk.dm-1.out‚ 57.92578125 disk.drbd2266.in‚0P‡2À2ü- _wandbruntime‚1466  _runtime‚1466ë²èÆýÀ & lr-AdamW/pg2‚0.0001670184222808256 trainer/global_step‚1899 $ _timestamp‚1.788417966546965e+09 _runtime‚1466.284515127 _step‚151 & lr-AdamW/pg1‚0.0001670184222808256<>_ËþÅ _runtime‚1466.284515127 _step‚151 & lr-AdamW/pg1‚0.0001670184222808256 & lr-AdamW/pg2‚0.0001670184222808256 trainer/global_step‚1899 $ _timestamp‚1.788417966546965e+09—$ð«Š2ÿ- _wandbruntime‚1466  _runtime‚1466w-Á,Ö'€ÐO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24973660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108132125083215 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109407978111488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110988189787648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110988864152064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788417966547676e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35666396 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110988864152064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1580886040576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119543228083712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1574415305947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15934652 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62985172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1580886040576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27318776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15934654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872­tÛrÛ'ÕO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1899 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24973660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106557038300160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24974102 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109407978111488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108132125083215 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35646360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109407978111488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚110988189787648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚110988864152064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109407960108544 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788417966547676e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35666396 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚110988864152064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121203431720448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106557055339520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1580886040576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1660399590400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚110988189787648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119543204143616 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27318774 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9039008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119543228083712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1574415305947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15934652 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1660227576832 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62985172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172013568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1580229679104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1580886040576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27318776 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27318774 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15934654 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121203627674112 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0˜‹ñ‹´2‚- _wandbruntime‚1466  _runtime‚14663|ƒÑMƒÛM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚153 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1661248327168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108187845648947 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚1899 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109464313147392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1581705481728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35664814 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884179672404795e+09 _runtime‚1466.978614083 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942822 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106611959470080 KEDeviceStatsMonitor.on_train_batchmQf”_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1661078424064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0qš¼æ&„àM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884179672404795e+09 _runtime‚1466.978614083 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121265933305344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106611942430720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1575231740911 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111046036632064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15942822 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106611959470080 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109464331150336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109464313147392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119604878821376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35684876 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9043569 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35684876 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62997604 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121266127148544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1661078424064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111046036632064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111045362254336 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63017668 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169903104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1581705481728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15942820 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109464331150336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24986851 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24986389 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119604854881280 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚153 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1661248327168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108187845648947 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚1899 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62997604 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27332792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109464313147392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1581049106944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1581705481728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63017668 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35664814 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35664814™p0á®2…- _wandbruntime‚1466  _runtime‚1466`¾M=X†Ò % _timestamp‚1.7884179672425725e+09 2loss/SRqqt4/assignment_loss‚1.6474618063885955 /loss/FRt4/detection_loss‚2.3131783248162736 .loss/FBt1/detection_loss‚13.44474464841187 trainer/global_step‚1899 0loss/FBt2/assignment_loss‚0.6736799809150398 0loss/FBt3/assignment_loss‚0.4817732510467181 /loss/FRt1/detection_loss‚2.3203956307591618 _runtime‚1466.980030556 /loss/FRt2/assignment_loss‚3.720617277677669 epoch‚1 _step‚154 2loss/SRqqt1/assignment_loss‚1.9706214719857924 1loss/SRqqt1/detection_loss‚2.3925316754059915 %loss/total_loss‚65.30319305401363 /loss/FRt1/assignment_loss‚4.600690000766033 /loss/FRt3/assignment_loss‚4.636280027031899 1loss/FBt4/assignment_loss‚0.20028505885472406 1loss/SRqqt2/detection_loss‚2.2442173610130944 .loss/FBt2/detection_loss‚3.426828258567386 .loss/FBt3/detection_loss‚4.509686162074407 /loss/FRt4/assignment_loss‚2.320777308707144 2loss/SRqqt2/assignment_loss‚1.6930774953216314 /loss/FRt2/detection_loss‚2.6286087091578993 0loss/SRqqt3/detection_loss‚2.557586584240198 1loss/SRqqt4/detection_loss‚1.2298993513082401 2loss/SRqqt3/assignment_loss‚2.0138070952147245 0loss/FBt1/assignment_loss‚0.6163408616557717 /loss/FRt3/detection_loss‚2.7493082016706465 /loss/FBt4/detection_loss‚0.9107965110227189§ÔŒU]‡× /loss/FRt4/assignment_loss‚2.320777308707144 2loss/SRqqt2/assignment_loss‚1.6930774953216314 /loss/FRt2/detection_loss‚2.6286087091578993 0loss/SRqqt3/detection_loss‚2.557586584240198 1loss/SRqqt4/detection_loss‚1.2298993513082401 2loss/SRqqt3/assignment_loss‚2.0138070952147245 0loss/FBt1/assignment_loss‚0.6163408616557717 /loss/FRt3/detection_loss‚2.7493082016706465 /loss/FBt4/detection_loss‚0.9107965110227189 % _timestamp‚1.7884179672425725e+09 2loss/SRqqt4/assignment_loss‚1.6474618063885955 /loss/FRt4/detection_loss‚2.3131783248162736 .loss/FBt1/detection_loss‚13.44474464841187 trainer/global_step‚1899 0loss/FBt2/assignment_loss‚0.6736799809150398 0loss/FBt3/assignment_loss‚0.4817732510467181 /loss/FRt1/detection_loss‚2.3203956307591618 _runtime‚1466.980030556 /loss/FRt2/assignment_loss‚3.720617277677669 epoch‚1 _step‚154 2loss/SRqqt1/assignment_loss‚1.9706214719857924 1loss/SRqqt1/detection_loss‚2.3925316754059915 %loss/total_loss‚65.30319305401363 /loss/FRt1/assignment_loss‚4.600690000766033 /loss/FRt3/assignment_loss‚4.636280027031899 1loss/FBt4/assignment_loss‚0.20028505885472406 1loss/SRqqt2/detection_loss‚2.2442173610130944 .loss/FBt2/detection_loss‚3.426828258567386 .loss/FBt3/detection_loss‚4.509686162074407šzí¬|’ˆ‚(:‡ ²¯äÔ¶‹Ÿ~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.746' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ_`‹G‰‚(:¼ ²¯äÔ½€¬€ disk.sda3.in‚0! proc.memory.rssMB‚ 2765.578125 cpu‚25.25851395293523 disk.drbd2080.in‚0 disk.dm-1.out‚ 58.6015625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚5690.31689453125 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚0* proc.memory.percent‚16.879749298095703 proc.cpu.threads‚17% disk./.usageGB‚253.88469314575195 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 0.7939453125 disk.sda.in‚0 disk.sda.out‚ 58.6015625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.09375 disk.drbd2266.in‚0$ memory_percent‚95.61316967010498( proc.memory.availableMB‚ 718.73828125' disk.nvme0c0n1.out‚5690.31689453125 disk.sda3.out‚ 58.6015625) disk./.usagePercent‚17.03841478045777 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.09375 disk.sda1.in‚0 disk.nvme0n1.in‚4.09375 network.sent‚ 3.242445e+06 network.recv‚612405" disk.dm-0.out‚5690.31689453125*Š‚(:’ Á¯äÔÂÏô} gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚164.968' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.993599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ö$çI‹‚(:¾ Á¯äÔ‘Îý disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.61324119567871 network.recv‚619530 disk.dm-0.in‚4.09375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 3.35065e+06 disk.dm-1.out‚ 59.07421875 disk.rbd1.in‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚5748.88330078125 disk.drbd2080.out‚0 disk.dm-1.in‚0 disk.sda.out‚ 59.07421875 disk.sda1.out‚0% disk.nvme0n1.out‚5748.88330078125! proc.memory.rssMB‚ 2765.578125* proc.memory.percent‚16.879749298095703 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 59.07421875 cpu‚25.25629140464215 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 718.7265625" disk.dm-0.out‚5748.88330078125 disk.sda.in‚0 disk.nvme0c0n1.in‚4.09375* disk./.usagePercent‚17.039040206863916% disk./.usageGB‚253.89401245117188 disk.rbd1.out‚ 0.85693359375.Îz’Œ‚(:‡ ЯäÔì¬Ï~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë3%#G‚(:¼ ЯäÔìÓ†disk.sda3.in‚0 disk.dm-0.in‚4.09375" disk.dm-0.out‚5833.26416015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0$ memory_percent‚95.61176300048828 disk.drbd2266.in‚0 disk.sda.out‚ 59.7109375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 network.sent‚ 3.357074e+06 disk.sda2.out‚0 disk.sda3.out‚ 59.7109375 disk.rbd1.in‚0 disk.rbd0.out‚0 network.recv‚624542% disk./.usageGB‚253.91846084594727 disk.dm-1.in‚0 disk.dm-1.out‚ 59.7109375 disk.nvme0c0n1.in‚4.09375* proc.memory.percent‚16.879749298095703 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0% disk.nvme0n1.out‚5833.26416015625' disk.nvme0c0n1.out‚5833.26416015625! proc.memory.rssMB‚ 2765.578125 cpu‚25.254236625147005 disk.rbd1.out‚ 0.85693359375 disk.sda1.in‚0% proc.memory.availableMB‚ 718.96875* disk./.usagePercent‚17.040680959150865 disk.drbd2266.out‚0 disk.sda.in‚0 disk.rbd0.in‚0=¢ò¨2Ž- _wandbruntime‚1503  _runtime‚1503NB±fÈ _runtime‚ 1503.21593754 _step‚155 ' lr-AdamW/pg1‚0.00016676885022711606 ' lr-AdamW/pg2‚0.00016676885022711606 trainer/global_step‚1949 % _timestamp‚1.7884180034783165e+09q®_XÍÇ ' lr-AdamW/pg2‚0.00016676885022711606 trainer/global_step‚1949 % _timestamp‚1.7884180034783165e+09 _runtime‚ 1503.21593754 _step‚155 ' lr-AdamW/pg1‚0.00016676885022711606›¥ÒÀ‹2‘- _wandbruntime‚1503  _runtime‚1503(V í×'’ÑO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36592448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1621871846400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113846827535872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112225630053888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122625741026816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615905083103 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚1949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28019576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124329221809664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _step‚156 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1503.216831457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110917495473515 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16343052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36572412 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112225612050944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884180034790475e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109302244828160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615250645355 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28019576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ÄÕ•À “ÖO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109302261867520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28019576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1703456842752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113847501900288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚467 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1703289023488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1621215484928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36592448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1621215484928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124329030050304 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28019574 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1621871846400 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113846827535872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112225630053888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28019574 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytesñ¸èª.large_pool.freed‚122625741026816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1615905083103 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚1949 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9268911 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28019576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36592448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124329221809664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _step‚156 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64591986 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1503.216831457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64612024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16343054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113847501900288 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122625764966912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110917495473515 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16343052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36572412 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113846827535872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9268446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110918166950623 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112225612050944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112225612050944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36572412 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1621871846400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884180034790475e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112225630053888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109302244828160 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1615250645355 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113904674380288 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 1503.92015193 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64624456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109357165998080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16351222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884180041820273e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1949 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124391559815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36590866 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚ 1503.92015193 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112281965089792 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64624456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9273046 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109357148958720 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16351220 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155223040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110973887516355 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113904674380288 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112281983092736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1622691287552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64644520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1704323274240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109357165998080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122687391764480 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16351222 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9273510 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112281983092736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64644520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28033592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36610928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884180041820273e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25624266 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚1949 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64624456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25624732 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124391559815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36590866 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1622691287552 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112281965089792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0M<ò2—- _wandbruntime‚1503  _runtime‚1503²/ºV˜Ð 0loss/SRqqt3/detection_loss‚2.343881825606028 /loss/FRt1/assignment_loss‚4.224528937869602 1loss/SRqqt2/assignment_loss‚2.266101293679741 1loss/SRqqt1/detection_loss‚2.4891418498009443 epoch‚1 _runtime‚1503.921053391 1loss/SRqqt3/assignment_loss‚2.311523192203962 0loss/FBt2/assignment_loss‚0.5696636283313637 0loss/FBt4/assignment_loss‚0.2459885242228059 /loss/FBt1/detection_loss‚2.0957302413880825 /loss/FBt4/detection_loss‚0.9204166717578968 2loss/SRqqt1/assignment_loss‚1.8276998167857528 0loss/FBt1/assignment_loss‚0.6007831981405616 /loss/FRt1/detection_loss‚2.3371677001317344 /loss/FRt2/detection_loss‚2.5831198409199714 _step‚158 .loss/FRt4/detection_loss‚2.462964995399765 /loss/FBt3/detection_loss‚3.2427908844417996 /loss/FRt4/assignment_loss‚2.495313786942026 2loss/SRqqt4/assignment_loss‚1.3187216154990657 1loss/SRqqt4/detection_loss‚1.0217781406176347 %loss/total_loss‚52.73536830392774 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚4.078360959887505 /loss/FRt3/assignment_loss‚4.444425330017552 0loss/SRqqt2/detection_loss‚2.370726010037793 /loss/FBt2/detection_loss‚3.2350162744522093 1loss/FBt3/assignment_loss‚0.37462548083729214 $ _timestamp‚1.788418004183599e+09 /loss/FRt3/detection_loss‚2.8748981049566558Žq n[™Õ /loss/FRt1/detection_loss‚2.3371677001317344 /loss/FRt2/detection_loss‚2.5831198409199714 _step‚158 .loss/FRt4/detection_loss‚2.462964995399765 /loss/FBt3/detection_loss‚3.2427908844417996 /loss/FRt4/assignment_loss‚2.495313786942026 2loss/SRqqt4/assignment_loss‚1.3187216154990657 1loss/SRqqt4/detection_loss‚1.0217781406176347 %loss/total_loss‚52.73536830392774 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚4.078360959887505 /loss/FRt3/assignment_loss‚4.444425330017552 0loss/SRqqt2/detection_loss‚2.370726010037793 /loss/FBt2/detection_loss‚3.2350162744522093 1loss/FBt3/assignment_loss‚0.37462548083729214 $ _timestamp‚1.788418004183599e+09 /loss/FRt3/detection_loss‚2.8748981049566558 0loss/SRqqt3/detection_loss‚2.343881825606028 /loss/FRt1/assignment_loss‚4.224528937869602 1loss/SRqqt2/assignment_loss‚2.266101293679741 1loss/SRqqt1/detection_loss‚2.4891418498009443 epoch‚1 _runtime‚1503.921053391 1loss/SRqqt3/assignment_loss‚2.311523192203962 0loss/FBt2/assignment_loss‚0.5696636283313637 0loss/FBt4/assignment_loss‚0.2459885242228059 /loss/FBt1/detection_loss‚2.0957302413880825 /loss/FBt4/detection_loss‚0.9204166717578968 2loss/SRqqt1/assignment_loss‚1.8276998167857528 0loss/FBt1/assignment_loss‚0.6007831981405616ž¿g”;œš‚(:‘ ߯äÔÖû‰~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚167.392' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.47839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰LàI›‚(:¾ ߯äÔ©¸˜€% disk.nvme0n1.out‚5886.68603515625 disk.rbd0.out‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2765.578125 disk.drbd2080.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚5886.68603515625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 60.32421875 disk.sda.out‚ 60.32421875 disk.dm-1.out‚ 60.32421875 disk.rbd1.in‚0$ memory_percent‚95.61257362365723 proc.cpu.threads‚17 network.recv‚628248% disk./.usageGB‚253.92177200317383 disk.sda1.in‚0" disk.dm-0.out‚5886.68603515625 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875 network.sent‚ 3.362432e+06* disk./.usagePercent‚17.040903173690587 disk.sda.in‚0' proc.memory.availableMB‚ 718.8359375 disk.drbd2265.out‚0 disk.dm-0.in‚4.09375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.09375 disk.rbd0.in‚0* proc.memory.percent‚16.879749298095703 disk.nvme0c0n1.in‚4.09375 disk.sda3.in‚0 disk.drbd2265.in‚0 cpu‚25.25226845270048n mî’œ‚(:‡ î¯äÔµÒŒ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼p7ÿF‚(:¾ î¯äÔç˨ disk.rbd1.out‚ 0.919921875% disk.nvme0n1.out‚5939.94189453125' proc.memory.availableMB‚ 718.6953125 cpu‚25.25028059588801 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2765.578125' disk.nvme0c0n1.out‚5939.94189453125 disk.rbd0.out‚0* disk./.usagePercent‚17.042358064507674 disk.sda3.out‚ 60.87890625 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 network.sent‚ 3.470826e+06% disk./.usageGB‚253.94345092773438 disk.drbd2080.in‚0 disk.drbd1716.in‚0$ memory_per8§¸cent‚95.61343193054199 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 60.87890625 disk.sda.in‚0 disk.sda.out‚ 60.87890625 disk.sda1.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.879749298095703 disk.nvme0c0n1.in‚4.09375 disk.sda3.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.09375 disk.drbd2266.in‚0 disk.sda1.in‚0 network.recv‚635697 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375" disk.dm-0.out‚5939.94189453125@æ¾×2ž- _wandbruntime‚1539  _runtime‚1539f-€~ÇŸÁ & lr-AdamW/pg1‚0.0001665021008129881 & lr-AdamW/pg2‚0.0001665021008129881 trainer/global_step‚1999 % _timestamp‚1.7884180402185133e+09 _runtime‚1539.956049371 _step‚159êš;öÌ Æ & lr-AdamW/pg2‚0.0001665021008129881 trainer/global_step‚1999 % _timestamp‚1.7884180402185133e+09 _runtime‚1539.956049371 _step‚159 & lr-AdamW/pg1‚0.0001665021008129881Ÿc(‚2¡- _wandbruntime‚1539  _runtime‚1539 “éz×'¢ÑO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125708301850112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37498464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115043263993344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884180402192962e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113704208818031 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16751452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 trainer/global_step‚1999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66238876 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1539.957116315 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚1607DäÜ'£ÖO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37498464 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115043263993344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9498795 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112047468395520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884180402192962e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28720376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9498353 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125708277910016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115043281996288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115043281996288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116705465284096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116706139648512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113704208818031 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66218838 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16751452 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113703537340923 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28720374 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1656740422511 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116705465284096 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127454729177600 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 trainer/global_step‚1999 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66238876 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚1539.957116315 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚112047451356160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37518500 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16751454 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169916416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1662201290752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37498464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1746451267584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125708301850112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 0ôïf2¤- _wandbruntime‚1540  _runtime‚1540GU¢/à&¥ÚM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112102355486720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚161 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37516918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚1540.644320541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28734392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788418040906216e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127517431830016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0©4zGû¦ßM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1663020718592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28734390 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66251308 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28734390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116763312128512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 QCDeviceStatsMonitor.o0¯ê!n_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112102355486720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚161 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1663020718592 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113759257906447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115099617032192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116763312128512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26262549 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9503379 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37516918 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37536980 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112102372526080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26263001 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66271372 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28734392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚1540.644320541 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66271372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28734392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16759622 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113759929383763 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 $ _timestamp‚1.788418040906216e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115099635035136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127517431830016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1657556857683 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125769928647680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1656902419727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¡Dƒv,2§- _wandbruntime‚1540  _runtime‚1540Ÿ+wX¨Ò epoch‚1 trainer/global_step‚1999 _runtime‚1540.645339484 2loss/SRqqt1/assignment_loss‚1.9156288771944887 1loss/FBt3/assignment_loss‚0.24232550181278806 .loss/FRt2/detection_loss‚2.573554062529614 %loss/total_loss‚49.27871880244449 % _timestamp‚1.7884180409078755e+09 /loss/FRt1/detection_loss‚2.4987052173718163 .loss/FRt3/detection_loss‚2.474324027697245 1loss/SRqqt1/detection_loss‚2.3149250246146145 1loss/SRqqt3/detection_loss‚1.9993680894374848 /loss/FBt1/detection_loss‚2.5284953032221114 /loss/FRt1/assignment_loss‚4.041527942470882 0loss/FRt4/assignment_loss‚2.1561116923888526 1loss/FBt1/assignment_loss‚0.47991935094562777 1loss/SRqqt4/detection_loss‚1.4616198558360338 -loss/FBt2/detection_loss‚3.19524884223938 /loss/FRt3/assignment_loss‚3.785111857785119 2loss/SRqqt4/assignment_loss‚1.5749362492295977 1loss/SRqqt2/assignment_loss‚1.837784958440204 1loss/FBt4/assignment_loss‚0.21334946746388336 _step‚162 /loss/FRt2/assignment_loss‚3.266013247402091 2loss/SRqqt3/assignment_loss‚1.9642663717269897 .loss/FRt4/detection_loss‚1.881659493347009 0loss/SRqqt2/detection_loss‚2.100625818551973 /loss/FBt3/detection_loss‚3.2229208025065335 /loss/FBt4/detection_loss‚1.0002872713708453 0loss/FBt2/assignment_loss‚0.5500094768593045r* ]©× /loss/FBt1/detection_loss‚2.5284953032221114 /loss/FRt1/assignment_loss‚4.041527942470882 0loss/FRt4/assignment_loss‚2.1561116923888526 1loss/FBt1/assignment_loss‚0.47991935094562777 1loss/SRqqt4/detection_loss‚1.4616198558360338 -loss/FBt2/detection_loss‚3.19524884223938 /loss/FRt3/assignment_loss‚3.785111857785119 2loss/SRqqt4/assignment_loss‚1.5749362492295977 1loss/SRqqt2/assignment_loss‚1.837784958440204 1loss/FBt4/assignment_loss‚0.21334946746388336 _step‚162 /loss/FRt2/assignment_loss‚3.266013247402091 2loss/SRqqt3/assignment_loss‚1.9642663717269897 .loss/FRt4/detection_loss‚1.881659493347009 0loss/SRqqt2/detection_loss‚2.100625818551973 /loss/FBt3/detection_loss‚3.2229208025065335 /loss/FBt4/detection_loss‚1.0002872713708453 0loss/FBt2/assignment_loss‚0.5500094768593045 epoch‚1 trainer/global_step‚1999 _runtime‚1540.645339484 2loss/SRqqt1/assignment_loss‚1.9156288771944887 1loss/FBt3/assignment_loss‚0.24232550181278806 .loss/FRt2/detection_loss‚2.573554062529614 %loss/total_loss‚49.27871880244449 % _timestamp‚1.7884180409078755e+09 /loss/FRt1/detection_loss‚2.4987052173718163 .loss/FRt3/detection_loss‚2.474324027697245 1loss/SRqqt1/detection_loss‚2.3149250246146145 1loss/SRqqt3/detection_loss‚1.9993680894374848¢œM©¿’ª‚(:‡ ý¯äÔƒŽš~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ók™-«‚(:¢ ý¯äÔŽ­§€! proc.memory.rssMB‚ 2765.578125# disk.nvme0c0n1.out‚ 5987.7578125 disk.sda2.out‚0 disk.sda3.out‚61.375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2080.in‚0 disk.dm-0.out‚ 5987.7578125 disk.drbd2266.in‚0$ disk./.usageGB‚254.0805549621582 disk.sda.in‚0 disk.sda.out‚61.375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.09375 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚4.09375 disk.nvme0n1.in‚4.09375* proc.memory.percent‚16.879749298095703 network.recv‚640065* disk./.usagePercent‚17.051559231295805 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚61.375 disk.drbd2266.out‚0 network.sent‚ 3.476792e+06 disk.sda1.out‚0 disk.drbd2265.in‚0$ memory_percent‚95.61395645141602& proc.memory.availableMB‚ 718.609375 cpu‚25.248402392446078 proc.cpu.threads‚17! disk.nvme0n1.out‚ 5987.7578125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875¢ˆ¿”¬‚(:’ Œ°äÔÛš‹~ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.085800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý÷&oG­‚(:¼ Œ°äÔɹé€ network.recv‚647688 disk.dm-1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚6063.13427734375* proc.memory.percent‚16.879749298095703 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚4.09375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.09375 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 cpu‚25.246351340522082$ disk./.usageGB‚254.0869483947754 disk.sda2.out‚0 disk.sda3.out‚ 61.88671875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 61.88671875$ memory_percent‚95.61395645141602 disk.sda2.in‚0 disk.dm-0.in‚4.09375" disk.dm-0.out‚6063.13427734375 network.sent‚ 3.585664e+06) disk./.usagePercent‚17.05198829992319 disk.drbd2080.in‚0 disk.dm-1.out‚ 61.88671875! proc.memory.rssMB‚ 2765.578125 disk.sda3.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6063.13427734375& proc.memory.availableMB‚ 718.609375M£C¾’®‚(:‡ ›°äÔú´±~ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 fiC¯‚(:¸ ›°äÔêê·€ disk.sda3.out‚ 62.453125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.244680779847204 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6110.80615234375 proc.cpu.threads‚17% disk./.usageGB‚254.10691833496094 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2765.578125 disk.dm-1.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚6110.80615234375$ memory_percent‚95.61738967895508" disk.dm-0.out‚6110.80615234375 disk.nvme0n1.in‚4.09375& proc.memory.availableMB‚ 718.046875 network.sent‚ 3.591046e+06 disk.dm-0.in‚4.09375 disk.rbd1.out‚ 0.919921875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 network.recv‚650913* disk./.usagePercent‚17.053328499364937 disk.sda3.in‚0 disk.dm-1.out‚ 62.453125 disk.sda.out‚ 62.453125 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.09375* proc.memory.percent‚16.879749298095703Ò|°2°- _wandbruntime‚1577  _runtime‚1577"èî&ȱ ' lr-AdamW/pg1‚0.00016621822962861982 ' lr-AdamW/pg2‚0.00016621822962861982 trainer/global_step‚2049 $ _timestamp‚1.788418077305457e+09 _runtime‚1577.043008509 _step‚163Ñ#T Í²Ç trainer/global_step‚2049 $ _timestamp‚1.788418077305457e+09 _runtime‚1577.043008509 _step‚163 ' lr-AdamW/pg1‚0.00016621822962861982 ' lr-AdamW/pg2‚0.00016621822962861982£¦ )Ë2³- _wandbruntime‚1577  _runtime‚1577«5—¯×'´ÑO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119564777396736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177079296 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119564777396736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38424516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17159854 _step‚164 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚2049 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884180773061566e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 _runtime‚1577.044057178 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚117860933938688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26888238 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116489579208331 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1703843458048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38424516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17159852 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚114792674923520 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29421176 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117860915935744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119564103032320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38424516 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1703187096576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117860933938688 TGDevicžŸæ_eeStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38444552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9727941 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580392377344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1696921324171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17159854 _step‚164 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚114792657884160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚2049 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119564103032320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7884180773061566e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153139200 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29421174 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38444552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67845690 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1703843458048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130580569456640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26887793 _runtime‚1577.044057178 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29421174 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9728384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1697575761919¤:’‚2¶- _wandbruntime‚1577  _runtime‚1577#îïFá&·ÛM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38442970 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117917268974592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26900854 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130642888802304 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1704662899200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 trainer/global_step‚2049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38442970 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114847562014720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚165 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119621949876736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29435192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119621949876736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29435190 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29435190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1704662899200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116545299773855 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884180780038126e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130643084742656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67898224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26900402 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29435190 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128852465530880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1698392197091 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1704662899200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38463032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚116545299773855 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9732832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195940352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117917286977536 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884180780038126e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117917286977536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9732382 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130643084742656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116545971251171 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67898224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128852489470976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 trainer/global_step‚2049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67898224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38442970 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67878160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚114847562014720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚165 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1704006524416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119621949876736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29435192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414¥ß>¯2¹- _wandbruntime‚1577  _runtime‚1577度±UºÏ 1loss/SRqqt4/detection_loss‚1.3594170704081252 _step‚166 /loss/FRt2/assignment_loss‚4.566130381364089 /loss/FRt4/assignment_loss‚2.072004414521731 2loss/SRqqt3/assignment_loss‚1.8512242208508884 1loss/FBt1/assignment_loss‚0.40270163633692846 0loss/FBt3/assignment_loss‚0.5753890888558494 2loss/SRqqt1/assignment_loss‚2.1696518381436665 epoch‚1 0loss/FBt4/assignment_loss‚0.3104111936060684 /loss/FRt1/assignment_loss‚4.729027731474056 0loss/SRqqt1/detection_loss‚2.221175891160965 /loss/FBt1/detection_loss‚1.2569566568078647 /loss/FRt4/detection_loss‚1.8347820135263295 1loss/SRqqt2/detection_loss‚2.8775571849611072 %loss/total_loss‚51.10142229253829 % _timestamp‚1.7884180780056198e+09 _runtime‚1577.743078637 /loss/FRt3/assignment_loss‚4.111210413051374 2loss/SRqqt4/assignment_loss‚1.3125866335766339 -loss/FRt1/detection_loss‚2.25328138401342 /loss/FRt3/detection_loss‚2.5316864306276496 0loss/SRqqt3/detection_loss‚2.005283093919941 /loss/FBt2/detection_loss‚3.1711830298105874 /loss/FBt4/detection_loss‚0.8969068565240861 -loss/FRt2/detection_loss‚2.78766721028548 /loss/FBt3/detection_loss‚2.8570676710870533 trainer/global_step‚2049 2loss/SRqqt2/assignment_loss‚2.1742362711164684 /loss/FBt2/assignment_loss‚0.773883976507932Åì4oZ»Ô 0loss/SRqqt3/detection_loss‚2.005283093919941 /loss/FBt2/detection_loss‚3.1711830298105874 /loss/FBt4/detection_loss‚0.8969068565240861 -loss/FRt2/detection_loss‚2.78766721028548 /loss/FBt3/detection_loss‚2.8570676710870533 trainer/global_step‚2049 2loss/SRqqt2/assignment_loss‚2.1742362711164684 /loss/FBt2/assignment_loss‚0.773883976507932 1loss/SRqqt4/detection_loss‚1.3594170704081252 _step‚166 /loss/FRt2/assignment_loss‚4.566130381364089 /loss/FRt4/assignment_loss‚2.072004414521731 2loss/SRqqt3/assignment_loss‚1.8512242208508884 1loss/FBt1/assignment_loss‚0.40270163633692846 0loss/FBt3/assignment_loss‚0.5753890888558494 2loss/SRqqt1/assignment_loss‚2.1696518381436665 epoch‚1 0loss/FBt4/assignment_loss‚0.3104111936060684 /loss/FRt1/assignment_loss‚4.729027731474056 0loss/SRqqt1/detection_loss‚2.221175891160965 /loss/FBt1/detection_loss‚1.2569566568078647 /loss/FRt4/detection_loss‚1.8347820135263295 1loss/SRqqt2/detection_loss‚2.8775571849611072 %loss/total_loss‚51.10142229253829 % _timestamp‚1.7884180780056198e+09 _runtime‚1577.743078637 /loss/FRt3/assignment_loss‚4.111210413051374 2loss/SRqqt4/assignment_loss‚1.3125866335766339 -loss/FRt1/detection_loss‚2.25328138401342 /loss/FRt3/detection_loss‚2.5316864306276496¦¯Ú¢³¼‚(:… ª°äÔÛí~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 "H½‚(:½ ª°äÔÜü¿€ disk.drbd2080.out‚0% disk./.usageGB‚254.10187530517578 disk.rbd1.out‚ 0.919921875 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚6159.02392578125 disk.drbd2266.in‚0& proc.memory.availableMB‚ 718.265625 disk.drbd2266.out‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚6159.02392578125 disk.drbd2265.out‚0 disk.drbd2080.in‚0$ memory_percent‚95.61605453491211 cpu‚25.242717075413886 network.recv‚655417 disk.sda.in‚0 disk.sda.out‚ 62.80859375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 62.80859375 disk.dm-1.in‚0 disk.dm-1.out‚ 62.80859375 network.sent‚ 3.597039e+06 disk.nvme0n1.in‚4.09375 disk.nvme0c0n1.in‚4.09375 disk.drbd2265.in‚0 disk.dm-0.in‚4.09375! proc.memory.rssMB‚ 2765.578125* proc.memory.percent‚16.879749298095703 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚6159.02392578125 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.05299005740467ˆqpƒ’¾‚(:‡ ¹°äÔ£âî} gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dNx=¿‚(:² ¹°äÔ¤–ó€! proc.memory.rssMB‚ 2765.578125 disk.sda.in‚0 disk.nvme0n1.in‚4.09375 disk.dm-0.out‚ 6206.3515625* disk./.usagePercent‚17.054513046225868 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375 network.recv‚662519 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 network.sent‚ 3.705301e+06% disk./.usageGB‚254.12456893920898 disk.sda1.out‚0 disk.dm-1.out‚ 63.47265625 disk.drbd2266.in‚0! disk.nvme0n1.out‚ 6206.3515625 cpu‚25.240778930654017 disk.rbd1.out‚ 0.919921875 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 63.47265625 disk.sda2.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.879749298095703 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.09375# disk.nvme0c0n1.out‚ 6206.3515625 disk.sda3.in‚0 disk.sda3.out‚ 63.47265625$ memory_percent‚95.61605453491211& proc.memory.availableMB‚ 718.265625:û´.2À- _wandbruntime‚1613  _runtime‚1613¦ƒ fÉÁà ' lr-AdamW/pg2‚0.00016591729583234052 trainer/global_step‚2099 % _timestamp‚1.7884181141935751e+09 _runtime‚1613.931095183 _step‚167 ' lr-AdamW/pg1‚0.00016591729583234052VÛ*—ÎÂÈ ' lr-AdamW/pg1‚0.00016591729583234052 ' lr-AdamW/pg2‚0.00016591729583234052 trainer/global_step‚2099 % _timestamp‚1.7884181141935751e+09 _runtime‚1613.931095183 _step‚167§NŒ-d2Ã- _wandbruntime‚1613  _runtime‚1613“Xèd¯ÄÑO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122423415144960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120678567878144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39370604 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122423415144960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 _step‚168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120678567878144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120678585881088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.curukQÿ(rent‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚1613.932177345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744172902400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120678585881088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1737756663579 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9958143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119276292552847 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131873375616512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884181141943614e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39350568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17568254 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1832696955392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30121976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚2099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39350568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119275621075739ƒ¥Ó\Ü'ÅÖO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7884181141943614e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39350568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17568254 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9957688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1832696955392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744172902400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30121976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122422740780544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 trainer/global_step‚2099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39350568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30121976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119275621075739 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122423415144960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120678567878144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39370604 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1738411101327 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚122423415144960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 _step‚168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120678567878144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚131873351676416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27525940 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120678585881088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1744829263872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚1613.932177345 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1744172902400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120678585881088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1737756663579 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30121974 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9958143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122422740780544 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39370604 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1744829263872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119276292552847 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚131873375616512¨â&¦Ä2Æ- _wandbruntime‚1614  _runtime‚1614³šá&ÇÛM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120734920916992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39369022 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27539128 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117592768542720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 trainer/global_step‚2099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133768559188992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833556774912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122479913247232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17576422 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17576422 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117592785582080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27538664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1744992330240 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30135992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17576420 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39369022 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27539128 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1738573098543 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117592768542720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69525076 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119331341641263 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1739227536499 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120734938919936 trainer/global_step‚2099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69505012 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚131935002414080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122480587624960 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39389084 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133768559188992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1745648705024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1833556774912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1744992330240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088©Š+2É- _wandbruntime‚1614  _runtime‚1614]'ÕšVÊÐ 0loss/FBt1/assignment_loss‚0.6351417005062103 /loss/FRt1/detection_loss‚2.2874090617353264 /loss/FBt1/detection_loss‚3.1948819875717165 /loss/FBt2/detection_loss‚3.0044078876574836 trainer/global_step‚2099 0loss/FBt3/assignment_loss‚0.6356594297619943 _step‚170 1loss/FBt4/assignment_loss‚0.21434430170555135 0loss/FRt1/assignment_loss‚3.9451511109417137 epoch‚1 % _timestamp‚1.7884181148840523e+09 2loss/SRqqt4/assignment_loss‚1.2660481872485032 /loss/FRt4/detection_loss‚1.9058893104749066 0loss/SRqqt1/detection_loss‚2.261059352984795 /loss/FRt2/assignment_loss‚4.226620416892202 1loss/SRqqt1/assignment_loss‚1.858169204913653 2loss/SRqqt2/assignment_loss‚2.3165764302015304 2loss/SRqqt3/assignment_loss‚1.6314822501902069 .loss/FBt2/assignment_loss‚0.62394372327041 1loss/SRqqt2/detection_loss‚2.3911378636956213 -loss/FBt4/detection_loss‚1.01694767309946 &loss/total_loss‚51.464810937037285 .loss/FRt2/detection_loss‚2.770308685930152 /loss/FRt3/detection_loss‚2.4847547041045295 /loss/FBt3/detection_loss‚3.3298475543657937 /loss/FRt3/assignment_loss‚4.430630694495307 /loss/FRt4/assignment_loss‚2.045188208775861 0loss/SRqqt3/detection_loss‚1.889943792883839 1loss/SRqqt4/detection_loss‚1.0992674036305161 _runtime‚1614.6215160498¤ïç[ËÕ epoch‚1 % _timestamp‚1.7884181148840523e+09 2loss/SRqqt4/assignment_loss‚1.2660481872485032 /loss/FRt4/detection_loss‚1.9058893104749066 0loss/SRqqt1/detection_loss‚2.261059352984795 /loss/FRt2/assignment_loss‚4.226620416892202 1loss/SRqqt1/assignment_loss‚1.858169204913653 2loss/SRqqt2/assignment_loss‚2.3165764302015304 2loss/SRqqt3/assignment_loss‚1.6314822501902069 .loss/FBt2/assignment_loss‚0.62394372327041 1loss/SRqqt2/detection_loss‚2.3911378636956213 -loss/FBt4/detection_loss‚1.01694767309946 &loss/total_loss‚51.464810937037285 .loss/FRt2/detection_loss‚2.770308685930152 /loss/FRt3/detection_loss‚2.4847547041045295 /loss/FBt3/detection_loss‚3.3298475543657937 /loss/FRt3/assignment_loss‚4.430630694495307 /loss/FRt4/assignment_loss‚2.045188208775861 0loss/SRqqt3/detection_loss‚1.889943792883839 1loss/SRqqt4/detection_loss‚1.0992674036305161 _runtime‚1614.621516049 0loss/FBt1/assignment_loss‚0.6351417005062103 /loss/FRt1/detection_loss‚2.2874090617353264 /loss/FBt1/detection_loss‚3.1948819875717165 /loss/FBt2/detection_loss‚3.0044078876574836 trainer/global_step‚2099 0loss/FBt3/assignment_loss‚0.6356594297619943 _step‚170 1loss/FBt4/assignment_loss‚0.21434430170555135 0loss/FRt1/assignment_loss‚3.9451511109417137ªŽ%RŠ’Ì‚(:‡ È°äÔ‘À»~ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2722 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð,‘XCÍ‚(:¸ È°äÔƒ´¬ disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2765.578125" disk./.usageGB‚254.13818359375 disk.dm-1.out‚ 63.9921875 disk.drbd1716.out‚0 disk.sda3.out‚ 63.9921875 network.recv‚667119 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.09375 disk.rbd1.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 718.1953125 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.09375$ memory_percent‚95.61648368835449* proc.memory.percent‚16.879749298095703 proc.cpu.threads‚17 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6264.02783203125 disk.sda3.in‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.239178242675738 network.sent‚ 3.711551e+06 disk.nvme0c0n1.in‚4.09375 disk.drbd2080.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚17.05542673711557" disk.dm-0.out‚6264.02783203125 disk.sda.out‚ 63.9921875% disk.nvme0n1.out‚6264.02783203125k’XD’΂(:‡ ×°äÔת‹~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 צnHÏ‚(:½ ×°äÔƒ¦€" disk.dm-0.out‚6317.46923828125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0$ disk./.usageGB‚254.1718521118164 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚6317.46923828125 disk.drbd2265.out‚0 cpu‚25.23720814200763 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0' proc.memory.availableMB‚ 718.1796875 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.09375 disk.dm-1.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 64.45703125 proc.cpu.threads‚17 network.sent‚ 3.819318e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-0.in‚4.09375 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 64.45703125 disk.sda1.out‚0% disk.nvme0n1.out‚6317.46923828125 network.recv‚673734 disk.dm-1.out‚ 64.45703125 disk.rbd1.out‚ 0.919921875 disk.drbd2265.in‚0$ memory_percent‚95.61657905578613! proc.memory.rssMB‚ 2765.578125* proc.memory.percent‚16.879749298095703* disk./.usagePercent‚17.0576862596127768‹‹’Ђ(:‡ æ°äÔâÙ~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6618 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 |AÑ‚(:¶ æ°äÔø•( proc.memory.availableMB‚ 718.18359375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 cpu‚25.23564811946517 network.recv‚678387( disk./.usagePercent‚17.0592842586415 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda.in‚0 disk.nvme0c0n1.in‚4.09375% disk.nvme0c0n1.out‚6380.080078125 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.09375# disk.nvme0n1.out‚6380.080078125 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 65.14453125 disk.rbd1.in‚0 network.sent‚ 3.82533e+06 disk.rbd0.out‚0 disk.dm-0.in‚4.09375 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2765.578125 disk.rbd0.in‚0 disk.sda3.out‚ 65.14453125* proc.memory.percent‚16.879749298095703% disk./.usageGB‚254.19566345214844 disk.sda3.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚6380.080078125 disk.sda.out‚ 65.14453125$ memory_percent‚95.61655521392822ï¢ÿ32Ò- _wandbruntime‚1650  _runtime‚1650²îÉXÉÓà _step‚171 ' lr-AdamW/pg1‚0.00016559936213830236 ' lr-AdamW/pg2‚0.00016559936213830236 trainer/global_step‚2149 % _timestamp‚1.7884181508496997e+09 _runtime‚1650.587253736â¦ÏÎÔÈ _runtime‚1650.587253736 _step‚171 ' lr-AdamW/pg1‚0.00016559936213830236 ' lr-AdamW/pg2‚0.00016559936213830236 trainer/global_step‚2149 % _timestamp‚1.7884181508496997e+09«c•)2Õ- _wandbruntime‚1650  _runtime‚1650»ß Ø'ÖÒO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136831949222912 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122061662943147 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1778592002987 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40296656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123496219820544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1785815069696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1875868903936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10187812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134955888559616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71119432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚172 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2149 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40276620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1785158708224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876036723200 _runtime‚1650.588414657 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1875868903936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17976654 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136831757463552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10187812 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17976652 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134955888559616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40276620 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123496237823488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71119432 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚172 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1785815069696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28164464 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30822776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125281378528768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40296656 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125282052893184 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30822776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1785158708224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2149 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40276620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1785158708224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125281378528768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28164917 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10188263 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122062334420255 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123496237823488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1779246440735 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30822774 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876036723200 _runtime‚1650.588414657 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71099394 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚173 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125339225373184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚1651.274469098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 trainer/global_step‚2149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788418151536376e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30836790 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785978136064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40295074fÅ7aç&ÚáM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136894447813632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125338550995456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30836792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10192350 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176194560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28177170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120337992110080 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200134656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122117383508671 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚125339225373184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10192812 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _runtime‚1651.274469098 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1785978136064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135017539297280 trainer/global_step‚2149 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30836790 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40315136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123552572859392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1786634510848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71151928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135017563237376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71131864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17984822 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚122118054985987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788418151536376e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40315136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30836790 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71151928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123552590862336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123552572859392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1785978136064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136894247678976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1876884576256 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40295074 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120337975070720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1780062875907 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125339225373184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚173 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17984820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28177634 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872­2ë 2Û- _wandbruntime‚1651  _runtime‚1651]•–OÜÉ /loss/FRt2/assignment_loss‚4.886117327958345 /loss/FRt3/assignment_loss‚3.734780971040117 0loss/FBt1/assignment_loss‚0.8931195742916316 0loss/FBt3/assignment_loss‚0.5308115653693676 .loss/FRt2/detection_loss‚2.887133851647377 trainer/global_step‚2149 $ _timestamp‚1.788418151538098e+09 _step‚174 .loss/FBt3/detection_loss‚3.153873860836029 /loss/FBt4/detection_loss‚1.0388395543592248 1loss/SRqqt4/assignment_loss‚1.490681206171469 0loss/FBt4/assignment_loss‚0.3367047493172232 1loss/SRqqt1/detection_loss‚2.0772748476929133 /loss/FRt4/assignment_loss‚2.463721642459648 1loss/SRqqt3/assignment_loss‚1.877639751881361 /loss/FBt2/assignment_loss‚0.684283495082387 /loss/FRt1/detection_loss‚2.3713650374036086 epoch‚1 _runtime‚ 1651.27551404 /loss/FRt1/assignment_loss‚4.108774831420497 -loss/FRt3/detection_loss‚2.55416155115087 /loss/SRqqt2/detection_loss‚2.18253739584576 1loss/SRqqt4/detection_loss‚1.0932291719108456 2loss/SRqqt1/assignment_loss‚1.8977430416478052 .loss/FRt4/detection_loss‚1.627709986075111 .loss/FBt1/detection_loss‚2.404725822309653 2loss/SRqqt2/assignment_loss‚2.2659087120132013 1loss/SRqqt3/detection_loss‚2.1842274841937153 /loss/FBt2/detection_loss‚3.4459814173834666 &loss/total_loss‚52.191346849461624nÒ{PTÝÎ /loss/FRt2/assignment_loss‚4.886117327958345 /loss/FRt3/assignment_loss‚3.734780971040117 0loss/FBt1/assignment_loss‚0.8931195742916316 0loss/FBt3/assignment_loss‚0.5308115653693676 .loss/FRt2/detection_loss‚2.887133851647377 trainer/global_step‚2149 $ _timestamp‚1.788418151538098e+09 _step‚174 .loss/FBt3/detection_loss‚3.153873860836029 /loss/FBt4/detection_loss‚1.0388395543592248 1loss/SRqqt4/assignment_loss‚1.490681206171469 0loss/FBt4/assignment_loss‚0.3367047493172232 1loss/SRqqt1/detection_loss‚2.0772748476929133 /loss/FRt4/assignment_loss‚2.463721642459648 1loss/SRqqt3/assignment_loss‚1.877639751881361 /loss/FBt2/assignment_loss‚0.684283495082387 /loss/FRt1/detection_loss‚2.3713650374036086 epoch‚1 _runtime‚ 1651.27551404 /loss/FRt1/assignment_loss‚4.108774831420497 -loss/FRt3/detection_loss‚2.55416155115087 /loss/SRqqt2/detection_loss‚2.18253739584576 1loss/SRqqt4/detection_loss‚1.0932291719108456 2loss/SRqqt1/assignment_loss‚1.8977430416478052 .loss/FRt4/detection_loss‚1.627709986075111 .loss/FBt1/detection_loss‚2.404725822309653 2loss/SRqqt2/assignment_loss‚2.2659087120132013 1loss/SRqqt3/detection_loss‚2.1842274841937153 /loss/FBt2/detection_loss‚3.4459814173834666 &loss/total_loss‚52.191346849461624®ù=pðÞ‚(:… õ°äÔéÝ¡~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05²¸¸Gß‚(:¼ õ°äÔï³ù€ disk.drbd2266.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.in‚0 disk.sda3.out‚ 65.78515625 network.sent‚ 3.831107e+06* disk./.usagePercent‚17.060509510700378 disk.sda.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚4.09375 disk.dm-1.out‚ 65.78515625 cpu‚25.233861067404664 disk.drbd2265.in‚0 disk.drbd2080.out‚0 network.recv‚682411 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.09375% disk.nvme0c0n1.out‚6413.603515625 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.dm-0.out‚6413.603515625 disk.dm-1.in‚0* proc.memory.percent‚16.879773139953613 disk.sda.out‚ 65.78515625# disk.nvme0n1.out‚6413.603515625 disk.sda2.in‚0( proc.memory.availableMB‚ 718.14453125% disk./.usageGB‚254.21392059326172 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0$ memory_percent‚95.61679363250732 disk.nvme0c0n1.in‚4.09375# proc.memory.rssMB‚ 2765.58203125 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda2.out‚0;Yà‚(:’ „±äÔ™Ô„~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.731200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`¯dAá‚(:¶ „±äÔ¶Âí€ disk.drbd2265.out‚0# proc.memory.rssMB‚ 2765.58203125) disk./.usagePercent‚17.06139760083666! disk.dm-0.out‚6474.8251953125 disk.rbd1.out‚ 0.919921875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚6474.8251953125 disk.dm-1.out‚ 66.265625 disk.drbd2266.in‚0 disk.rbd1.in‚0# memory_percent‚95.6153392791748 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 network.sent‚ 3.939694e+06 network.recv‚690100$ disk.nvme0n1.out‚6474.8251953125 disk.nvme0c0n1.in‚4.09375 disk.drbd2080.out‚0 cpu‚25.232393978830064 disk.drbd2265.in‚0* proc.memory.percent‚16.879773139953613 disk.sda.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚0 disk.dm-0.in‚4.09375 disk.sda.out‚ 66.265625 disk.sda3.out‚ 66.265625' proc.memory.availableMB‚ 718.3828125% disk./.usageGB‚254.22715377807617 disk.sda1.in‚0 disk.sda2.in‚0#2â- _wandbruntime‚1687  _runtime‚16873ȆdÇãÁ & lr-AdamW/pg1‚0.0001652644948034106 & lr-AdamW/pg2‚0.0001652644948034106 trainer/global_step‚2199 % _timestamp‚1.7884181875657208e+09 _runtime‚1687.303268048 _step‚175áø«ÌäÆ & lr-AdamW/pg1‚0.0001652644948034106 & lr-AdamW/pg2‚0.0001652644948034106 trainer/global_step‚2199 % _timestamp‚1.7884181875657208e+09 _runtime‚1687.303268048 _step‚175¯üÂ{92å- _wandbruntime‚1687  _runtime‚1687<œKjoæÓO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18385052 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1919046320640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123028294507520 trainer/global_step‚2199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126313871762944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826144514048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138038425442816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176207872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139957495703552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28800887 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10415391 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41202672 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚126313889765888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28800443 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72726246 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31523576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41222708 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41202672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1820081780143 _runtime‚1687.304220045 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884181875664446e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31523574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1819427342395 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826800875520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 UADevice¡y jStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826144514048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚176 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18385054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124847704810555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126313889765888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126313871762944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128140016276992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18385054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚124847704810555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31523574 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126313889765888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139957295555584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126313871762944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884181875664446e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31523574 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123028277468160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1819427342395 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826800875520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72746284 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10415833 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128140690641408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128140016276992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826144514048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚176 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124904096853395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128197188743680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10420405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31537590 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126370224801792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0êW 9€êâM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 _step‚177 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126370242804736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827620316672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140019996728320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚1687.998947586 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72758716 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28813627 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18393222 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41221126 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128197188743680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126370242804736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41241188 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128197863121408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31537592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919896607744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72778780 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140019809176576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10420405 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827620316672 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚124903425376079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123083198638080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1820898215315 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72778780 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31537590 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126370224801792 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826963941888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138100076180480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10419968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123083181598720 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1919732996096 trainer/global_step‚2199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41221126 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1820243777359 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72758716 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28813188 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚124904096853395 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18393220 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126370224801792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41241188 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29439617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 _step‚180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867786681344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1867786681344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141120962326016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867130319872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32224374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42148760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29439617 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10646165 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129131523705344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32224376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18793452 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2249 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129131523705344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143083024892416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1961881292800 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42128724 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 _step‚180 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚130998654025216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867786681344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚130998654025216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141120986266112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125773483996160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32224374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143082843618816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127633746677963 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74353098 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1867786681344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18793454 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42147178 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 ^JDeviceStatsMonitor.on_train_batch 1ëæO_end/requested_bytes.large_pool.allocated‚125828405166080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1867949747712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884182250711963e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141182637003776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720õœ"3è&úâM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125828388126720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141182613063680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10650700 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127690138720803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861733554723 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1867949747712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127689467243487 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42147178 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42147178 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131056500869632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181260288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29452320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131055826491904 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10651153 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125828405166080 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129187894747136 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18801620 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74405632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32238390 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1867949747712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884182250711963e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143145546705408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143145365445120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74405632 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32238390 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29452775 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141182637003776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1962752381440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32238392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1868606122496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1962909701632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18801622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74385568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360µqo‚Ž2û- _wandbruntime‚1724  _runtime‚1724´´Ã?\üÖ /loss/FRt3/assignment_loss‚4.132293764352799 2loss/SRqqt1/assignment_loss‚2.5725128267492567 /loss/FRt1/assignment_loss‚4.255301828287085 2loss/SRqqt2/assignment_loss‚2.3411648031827568 1loss/SRqqt3/detection_loss‚1.9669323002590853 0loss/FRt4/assignment_loss‚1.9943348101849825 /loss/FBt1/detection_loss‚2.8445980158867314 .loss/FBt3/detection_loss‚3.171372135480245 &loss/total_loss‚52.322080127619785 % _timestamp‚1.7884182250730052e+09 0loss/FBt3/assignment_loss‚0.5328018826742967 /loss/FRt1/detection_loss‚3.0518265728439604 /loss/FBt4/detection_loss‚0.8648746934377899 _step‚182 2loss/SRqqt4/assignment_loss‚1.1109621683725073 0loss/FBt1/assignment_loss‚0.6047700490587538 1loss/FBt4/assignment_loss‚0.33285471847688314 .loss/FRt2/detection_loss‚2.907629070743438 epoch‚1 trainer/global_step‚2249 2loss/SRqqt3/assignment_loss‚1.5253596643314642 /loss/FRt3/detection_loss‚2.1319790828227996 1loss/SRqqt4/detection_loss‚1.2146301462633322 /loss/FBt2/detection_loss‚3.7087301782199313 0loss/FRt2/assignment_loss‚4.1094813962136545 .loss/FRt4/detection_loss‚2.032905030644165 0loss/FBt2/assignment_loss‚0.4587796476490829 1loss/SRqqt1/detection_loss‚2.1098181846595945 1loss/SRqqt2/detection_loss‚2.3461671568251945 _runtime‚1724.8104696961ŽpQaýÛ 2loss/SRqqt3/assignment_loss‚1.5253596643314642 /loss/FRt3/detection_loss‚2.1319790828227996 1loss/SRqqt4/detection_loss‚1.2146301462633322 /loss/FBt2/detection_loss‚3.7087301782199313 0loss/FRt2/assignment_loss‚4.1094813962136545 .loss/FRt4/detection_loss‚2.032905030644165 0loss/FBt2/assignment_loss‚0.4587796476490829 1loss/SRqqt1/detection_loss‚2.1098181846595945 1loss/SRqqt2/detection_loss‚2.3461671568251945 _runtime‚1724.810469696 /loss/FRt3/assignment_loss‚4.132293764352799 2loss/SRqqt1/assignment_loss‚2.5725128267492567 /loss/FRt1/assignment_loss‚4.255301828287085 2loss/SRqqt2/assignment_loss‚2.3411648031827568 1loss/SRqqt3/detection_loss‚1.9669323002590853 0loss/FRt4/assignment_loss‚1.9943348101849825 /loss/FBt1/detection_loss‚2.8445980158867314 .loss/FBt3/detection_loss‚3.171372135480245 &loss/total_loss‚52.322080127619785 % _timestamp‚1.7884182250730052e+09 0loss/FBt3/assignment_loss‚0.5328018826742967 /loss/FRt1/detection_loss‚3.0518265728439604 /loss/FBt4/detection_loss‚0.8648746934377899 _step‚182 2loss/SRqqt4/assignment_loss‚1.1109621683725073 0loss/FBt1/assignment_loss‚0.6047700490587538 1loss/FBt4/assignment_loss‚0.33285471847688314 .loss/FRt2/detection_loss‚2.907629070743438 epoch‚1 trainer/global_step‚2249¶ô ¾Ò‘þ‚(:† ±±äÔ™Œ~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.175' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jæñŠHÿ‚(:½ ±±äÔý¶þ€ disk.sda1.out‚0 disk.nvme0c0n1.in‚4.09375 disk.sda2.out‚0 disk.sda3.out‚ 67.88671875 disk.drbd2265.in‚0 disk.drbd2080.out‚0! disk.dm-0.out‚6629.6923828125 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.drbd2080.in‚0 disk.sda.in‚0 disk.sda2.in‚0$ memory_percent‚95.61975002288818* disk./.usagePercent‚17.063479198097564 disk.rbd1.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2765.58984375* proc.memory.percent‚16.879820823669434 disk.dm-0.in‚4.09375 disk.dm-1.in‚0 disk.dm-1.out‚ 67.88671875 disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 67.88671875 disk.drbd1716.out‚0$ disk.nvme0n1.out‚6629.6923828125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚6629.6923828125 network.recv‚704853 disk.sda1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 717.66015625 cpu‚25.2277030252685 proc.cpu.threads‚17 network.sent‚ 4.059363e+06 disk.drbd2266.in‚0 disk.sda3.in‚0% disk./.usageGB‚254.25817108154297 disk.drbd2266.out‚0à{’€‚(:‡ À±äÔæÌ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…„ÚM‚(: À±äÔ¡œÄ network.recv‚712725 disk.sda1.out‚0% disk.nvme0n1.out‚6690.37939453125 disk.sda3.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17% disk./.usageGB‚254.27130508422852 disk.drbd2080.out‚0$ memory_percent‚95.61846256256104( proc.memory.availableMB‚ 717.87109375* proc.memory.percent‚16.879820823669434 network.sent‚ 4.168034e+06* disk./.usagePercent‚17.064360632037953 disk.drbd2080.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 68.52734375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2765.58984375 disk.dm-0.in‚4.09375 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.09375 disk.rbd0.in‚0 disk.sda2.out‚0" disk.dm-0.out‚6690.37939453125 disk.sda.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚6690.37939453125 cpu‚25.226176895472047 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 68.52734375 disk.sda.out‚ 68.52734375 disk.nvme0c0n1.in‚4.09375”ž†×‚‚(:’ ϱäÔÜä˜~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.259' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.251800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠBm^Dƒ‚(:¹ ϱäÔ澊€ disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚4.09375 disk.rbd1.in‚0$ disk.nvme0n1.out‚6743.3017578125 disk.nvme0c0n1.in‚4.09375 network.recv‚715480 disk.sda3.out‚ 69.2421875 network.sent‚ 4.172346e+06 disk.drbd2265.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚4.09375 disk.drbd2080.out‚0 disk.dm-1.out‚ 69.2421875 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 69.2421875 disk.sda1.out‚0$ memory_percent‚95.61853408813477) disk./.usagePercent‚17.06475206755781 disk.drbd2080.in‚0 disk.drbd1716.in‚0& proc.memory.availableMB‚ 717.859375# proc.memory.rssMB‚ 2765.58984375* proc.memory.percent‚16.879820823669434! disk.dm-0.out‚6743.3017578125 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.224595968948332 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚6743.3017578125% disk./.usageGB‚254.27713775634766 disk.drbd2266.in‚0mÐœ2„- _wandbruntime‚1760  _runtime‚1760IoFkÉ…Ã % _timestamp‚1.7884182608749387e+09 _runtime‚1760.612449104 _step‚183 ' lr-AdamW/pg1‚0.00016454424186887112 ' lr-AdamW/pg2‚0.00016454424186887112 trainer/global_step‚2299¬Š·^Î†È trainer/global_step‚2299 % _timestamp‚1.7884182608749387e+09 _runtime‚1760.612449104 _step‚183 ' lr-AdamW/pg1‚0.00016454424186887112 ' lr-AdamW/pg2‚0.00016454424186887112·dIÖ2‡- _wandbruntime‚1760  _runtime‚1760Aü pÖ!ˆÓO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2005010566656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884182608755915e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1908772487168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1908772487168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144203523149312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131949175647744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 trainer/global_step‚2299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144203499209216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32925176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130419788545371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146208341956608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43074812 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128518707563520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128518690524160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_starJõ†ðt/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚184 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1901098021211 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2004842747392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32925174 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1908116125696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191759360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144203523149312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131949175647744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30076361 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 trainer/global_step‚2299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75999988 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32925174 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144203499209216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75979950 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32925176 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚130419788545371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131949175647744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30076801 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43054776 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146208341956608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43074812 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133857291773440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43074812 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43054776 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131949193650688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128518707563520 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128518690524160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167819264 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1901752458959 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10874509 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚130420460022479 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚184 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1901098021211 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133857966137856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2004842747392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19201854 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884182608755915e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1908772487168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133857966137856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19201852 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131949193650688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1908116125696 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133857291773440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1908772487168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚1761.322169343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32939192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1909591928320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32939192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19210020 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144265173886976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1902568894131 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76032484 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133915138617856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144265149946880 % _timestamp‚1.7884182615842388e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146270850155520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 trainer/global_step‚2299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43073230 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10879482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133915138617856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19210022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2005865917440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128573611694080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800‚†ëYè&ŒâM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130476180588211 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133915138617856 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76032484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32939190 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19210022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2005865917440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128573611694080 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32939190 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132005546689536 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30089504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10879025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2005700208640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚1761.322169343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43093292 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32939192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1909591928320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146271039804416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32939192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19210020 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚144265173886976 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1902568894131 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76032484 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76012420 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133915138617856 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144265149946880 % _timestamp‚1.7884182615842388e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146270850155520 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚130475509110895 trainer/global_step‚2299 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43073230 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132005528686592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1901914456175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43093292 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76012420 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43073230 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚185 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30089045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10879482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¹°Z 2- _wandbruntime‚1761  _runtime‚1761Sóü¿ŽÍ /loss/FRt1/detection_loss‚2.6190216015366947 /loss/FRt3/assignment_loss‚3.908010416305982 1loss/SRqqt2/assignment_loss‚2.495575633280131 1loss/SRqqt4/detection_loss‚1.0814090077520202 %loss/total_loss‚52.48043124577624 epoch‚1 _step‚186 2loss/SRqqt4/assignment_loss‚1.3050909178993224 0loss/FBt3/assignment_loss‚0.2043813600515326 .loss/FRt2/detection_loss‚2.578273813391841 /loss/FBt4/detection_loss‚1.0800537905569685 % _timestamp‚1.7884182615858274e+09 _runtime‚1761.323255433 0loss/SRqqt2/detection_loss‚2.141849163843661 .loss/FBt1/detection_loss‚2.966059930049456 trainer/global_step‚2299 .loss/FRt3/detection_loss‚2.349563400332744 0loss/SRqqt3/detection_loss‚2.198974686861038 /loss/FRt1/assignment_loss‚4.042940385201398 0loss/FRt4/assignment_loss‚1.9262172480424244 1loss/SRqqt1/assignment_loss‚2.125262722775743 0loss/FBt2/assignment_loss‚0.708451297¼u”8205263 /loss/FRt4/detection_loss‚1.9969200193881989 /loss/FBt1/assignment_loss‚0.683580306860095 1loss/FBt4/assignment_loss‚0.21941324674805732 1loss/SRqqt1/detection_loss‚2.3604038099984863 /loss/FRt2/assignment_loss‚4.880794070487799 2loss/SRqqt3/assignment_loss‚1.5553894385695457 /loss/FBt2/detection_loss‚2.8787968208392463 -loss/FBt3/detection_loss‚4.17399815718333ÄýîùXÒ /loss/FRt1/assignment_loss‚4.042940385201398 0loss/FRt4/assignment_loss‚1.9262172480424244 1loss/SRqqt1/assignment_loss‚2.125262722775743 0loss/FBt2/assignment_loss‚0.7084512978205263 /loss/FRt4/detection_loss‚1.9969200193881989 /loss/FBt1/assignment_loss‚0.683580306860095 1loss/FBt4/assignment_loss‚0.21941324674805732 1loss/SRqqt1/detection_loss‚2.3604038099984863 /loss/FRt2/assignment_loss‚4.880794070487799 2loss/SRqqt3/assignment_loss‚1.5553894385695457 /loss/FBt2/detection_loss‚2.8787968208392463 -loss/FBt3/detection_loss‚4.17399815718333 /loss/FRt1/detection_loss‚2.6190216015366947 /loss/FRt3/assignment_loss‚3.908010416305982 1loss/SRqqt2/assignment_loss‚2.495575633280131 1loss/SRqqt4/detection_loss‚1.0814090077520202 %loss/total_loss‚52.48043124577624 epoch‚1 _step‚186 2loss/SRqqt4/assignment_loss‚1.3050909178993224 0loss/FBt3/assignment_loss‚0.2043813600515326 .loss/FRt2/detection_loss‚2.578273813391841 /loss/FBt4/detection_loss‚1.0800537905569685 % _timestamp‚1.7884182615858274e+09 _runtime‚1761.323255433 0loss/SRqqt2/detection_loss‚2.141849163843661 .loss/FBt1/detection_loss‚2.966059930049456 trainer/global_step‚2299 .loss/FRt3/detection_loss‚2.349563400332744 0loss/SRqqt3/detection_loss‚2.198974686861038ºí‹›’‚(:‡ Þ±äÔ²êæ~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.834' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢IØ4F‘‚(:» Þ±äÔâó¢) disk./.usagePercent‚17.06618545374202 disk.dm-0.in‚4.09375 disk.sda.in‚0 disk.nvme0n1.in‚4.09375 disk.sda3.in‚0 cpu‚25.223242472010103 disk.sda2.in‚0 network.recv‚720210 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.879820823669434" disk.dm-0.out‚6815.58740234375 disk.dm-1.out‚ 69.8203125 disk.sda1.out‚0' disk.nvme0c0n1.out‚6815.58740234375 disk.drbd2265.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2765.58984375 disk.sda2.out‚0& proc.memory.availableMB‚ 717.546875 disk.sda3.out‚ 69.8203125$ disk./.usageGB‚254.2984962463379 disk.sda.out‚ 69.8203125 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.09375 disk.drbd2265.in‚0$ memory_percent‚95.62044143676758 proc.cpu.threads‚17 network.sent‚ 4.178326e+06 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875% disk.nvme0n1.out‚6815.58740234375!3ÌÚ’‚(:’ í±äÔžèš~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚124.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.973200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹\ù¦@“‚(:µ í±äÔÙþŽ€% disk./.usageGB‚254.28744506835938 disk.dm-0.out‚6865.185546875 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.dm-0.in‚4.09375 disk.dm-1.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 70.390625 disk.drbd1716.out‚0$ memory_percent‚95.61896324157715 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.09375# proc.memory.rssMB‚ 2765.58984375* proc.memory.percent‚16.879820823669434 network.sent‚ 4.286462e+06 disk.sda2.out‚0# disk.nvme0n1.out‚6865.185546875 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚17.065443799915325 disk.sda3.in‚0 disk.sda3.out‚ 70.390625 disk.dm-1.out‚ 70.390625' proc.memory.availableMB‚ 717.7890625 network.recv‚727131 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚6865.185546875 disk.drbd2266.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚4.09375 cpu‚25.221702923737787Áo°2”- _wandbruntime‚1797  _runtime‚1797šüeTÈ•Â ' lr-AdamW/pg1‚0.00016415900636885546 ' lr-AdamW/pg2‚0.00016415900636885546 trainer/global_step‚2349 $ _timestamp‚1.788418297907737e+09 _runtime‚1797.645288938 _step‚187­0Í–Ç _step‚187 ' lr-AdamW/pg1‚0.00016415900636885546 ' lr-AdamW/pg2‚0.00016415900636885546 trainer/global_step‚2349 $ _timestamp‚1.788418297907737e+09 _runtime‚1797.645288938»µ[ú2—- _wandbruntime‚1797  _runtime‚1797Ç›ŠØ'˜ÒO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134766827590144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149333833541120 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚188 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2047797448704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11104624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43980828 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33625974 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136716603886080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133206501889887 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133205830412779 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77606802>É`Ý'™×O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30714878 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131263897052160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚188 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149334031591936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131263914091520 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134766845593088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147286036092416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2047971559424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2047797448704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11104624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1949101931520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1949758292992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33625974 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43980828 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33625974 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585024 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚136716603886080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133206501889887 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198050816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133205830412779 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44000864 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43980828 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136715929521664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77606802 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77626840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134766827590144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149333833541120 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 % _timestamp‚1.7884182986081045e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43999282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2048657634816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2048821246464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor µ Æ .on_train_batch_end/inactive_split.small_pool.allocated‚11109152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944Ä.!ƒè&œâM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136773776366080 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77659336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2048821246464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133262222455619 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30727125 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30727574 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131318818222080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77639272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚2349 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656374784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77639272 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11109152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33639992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33639990 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33639992 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44019344 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43999282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149396532016640 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136773101988352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1942749795583 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134823198631936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1949921359360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656374784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136773776366080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133261550978303 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1943404233539 % _timestamp‚1.7884182986081045e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43999282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147347686830080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19618420 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11108705 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134823180628992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131318801182720 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134823198631936 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2048657634816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844½|@]r2- _wandbruntime‚1798  _runtime‚1798™l¼UžÏ /loss/FRt1/detection_loss‚2.3641208277808294 .loss/FRt2/detection_loss‚2.606321470439434 .loss/FBt2/detection_loss‚2.864600121974945 /loss/FBt3/detection_loss‚2.7047944515943527 /loss/FBt4/detection_loss‚0.9530997895113508 %loss/total_loss‚49.25061070361676 1loss/SRqqt2/detection_loss‚2.1821588029464087 % _timestamp‚1.7884182986096058e+09 _runtime‚ 1798.34705578 /loss/FRt1/assignment_loss‚4.323742558558782 0loss/FBt2/assignment_loss‚0.5460603443017362 .loss/FBt1/detection_loss‚1.988446248529343 _step‚190 0loss/FRt2/assignment_loss‚3.1159220427274703 trainer/global_step‚2349 0loss/FBt1/assignment_loss‚0.6788442490185914 /loss/FRt3/assignment_loss‚3.923709827992651 0loss/FBt4/assignment_loss‚0.2823856939856455 /loss/FRt4/detection_loss‚1.7762343012900264 1loss/SRqqt3/detection_loss‚2.3917376407318645 1loss/SRqqt1/assignment_loss‚2.138159649570783 1loss/SRqqt2/assignment_loss‚2.119291127038499 .loss/FRt3/detection_loss‚2.603362328476376 0loss/SRqqt1/detection_loss‚2.448476372162501 epoch‚1 0loss/FRt4/assignment_loss‚2.2335605919361115 2loss/SRqqt3/assignment_loss‚1.7900954460104306 2loss/SRqqt4/assignment_loss‚1.3163633052958175 0loss/FBt3/assignment_loss‚0.7482591271400452 1loss/SRqqt4/detection_loss‚1.1508643846027553•ÍZŸÔ _runtime‚ 1798.34705578 /loss/FRt1/assignment_loss‚4.323742558558782 0loss/FBt2/assignment_loss‚0.5460603443017362 .loss/FBt1/detection_loss‚1.988446248529343 _step‚190 0loss/FRt2/assignment_loss‚3.1159220427274703 trainer/global_step‚2349 0loss/FBt1/assignment_loss‚0.6788442490185914 /loss/FRt3/assignment_loss‚3.923709827992651 0loss/FBt4/assignment_loss‚0.2823856939856455 /loss/FRt4/detection_loss‚1.7762343012900264 1loss/SRqqt3/detection_loss‚2.3917376407318645 1loss/SRqqt1/assignment_loss‚2.138159649570783 1loss/SRqqt2/assignment_loss‚2.119291127038499 .loss/FRt3/detection_loss‚2.603362328476376 0loss/SRqqt1/detection_loss‚2.448476372162501 epoch‚1 0loss/FRt4/assignment_loss‚2.2335605919361115 2loss/SRqqt3/assignment_loss‚1.7900954460104306 2loss/SRqqt4/assignment_loss‚1.3163633052958175 0loss/FBt3/assignment_loss‚0.7482591271400452 1loss/SRqqt4/detection_loss‚1.1508643846027553 /loss/FRt1/detection_loss‚2.3641208277808294 .loss/FRt2/detection_loss‚2.606321470439434 .loss/FBt2/detection_loss‚2.864600121974945 /loss/FBt3/detection_loss‚2.7047944515943527 /loss/FBt4/detection_loss‚0.9530997895113508 %loss/total_loss‚49.25061070361676 1loss/SRqqt2/detection_loss‚2.1821588029464087 % _timestamp‚1.7884182986096058e+09¾sñ9k’ ‚(:‡ ü±äÔœ™Ù gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à´¡Úa¡‚(:Ö ü±äÔáÇÖ€% disk./.usageGB‚254.31418228149414 disk.drbd2265.in‚0 disk.rbd1.out‚ 0.919921875 disk.nvme0n1.in‚4.09375 network.recv‚732028 disk.sda2.out‚0 disk.sda3.in‚ 62.9296875 disk.drbd2080.in‚0 disk.dm-1.in‚ 62.9296875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 4.292854e+06 disk.rbd0.out‚0& disk.nvme0c0n1.out‚6927.1591796875 disk.sda.in‚ 62.9296875 disk.drbd1716.in‚0( proc.memory.availableMB‚ 717.77734375 cpu‚25.220276337429457 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2765.58984375* disk./.usagePercent‚17.067238156722816$ disk.nvme0n1.out‚6927.1591796875$ memory_percent‚95.61903476715088 disk.sda2.in‚0 disk.sda3.out‚ 88.703125 disk.dm-0.in‚4.09375! disk.dm-0.out‚6927.1591796875 disk.nvme0c0n1.in‚4.09375 disk.dm-1.out‚ 88.703125 disk.sda.out‚ 88.69921875* proc.memory.percent‚16.879820823669434cX®’¢‚(:‡ ‹²äÔä¤~ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ó6m£‚(:â ‹²äÔô»ž disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.dm-1.out‚ 136.53515625 disk.drbd2266.in‚0 disk.sda.out‚ 136.53515625 disk.sda1.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.068009763431018 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.09375! disk.dm-0.out‚6977.0107421875( proc.memory.availableMB‚ 717.73046875 disk.drbd2080.out‚0 network.sent‚ 4.400441e+06% disk./.usageGB‚254.32567977905273 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2765.58984375* proc.memory.percent‚16.879820823669434 disk.rbd1.out‚ 0.919921875 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.218737599367294 network.recv‚738175 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375# memory_percent‚95.6193208694458$ disk.nvme0n1.out‚6977.0107421875 disk.nvme0c0n1.in‚4.09375& disk.nvme0c0n1.out‚6977.0107421875 disk.sda3.out‚ 136.53515625 disk.nvme0n1.in‚4.09375XyQ”¤‚(:’ š²äÔõܧ~ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„‹Åp¥‚(:å š²äÔÈ¿£ disk.nvme0c0n1.in‚4.09375' disk.nvme0c0n1.out‚7040.19189453125$ memory_percent‚95.61936855316162 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 199.86328125 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.919921875 disk.sda.out‚ 199.86328125 disk.drbd1716.in‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚7040.19189453125# proc.memory.rssMB‚ 2765.58984375 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17% disk./.usageGB‚254.34183883666992 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.09375" disk.dm-0.out‚7040.19189453125 disk.dm-0.in‚4.09375 disk.rbd0.in‚0( proc.memory.availableMB‚ 717.72265625 cpu‚25.217372850157716 disk.drbd2265.out‚0 disk.dm-1.out‚ 199.86328125 disk.drbd2266.in‚0 disk.rbd1.in‚0 network.sent‚ 4.406512e+06 network.recv‚742758) disk./.usagePercent‚17.06909421134606 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.879820823669434 disk.sda2.in‚0 disk.sda2.out‚0ìö Ý2¦- _wandbruntime‚1834  _runtime‚1834OS‹êȧ % _timestamp‚1.7884183346298501e+09 _runtime‚ 1834.36734429 _step‚191 ' lr-AdamW/pg1‚0.00016375713739596968 ' lr-AdamW/pg2‚0.00016375713739596968 trainer/global_step‚2399W®>ŸÍ¨Ç ' lr-AdamW/pg1‚0.00016375713739596968 ' lr-AdamW/pg2‚0.00016375713739596968 trainer/global_step‚2399 % _timestamp‚1.7884183346298501e+09 _runtime‚ 1834.36734429 _step‚191¿ Z€f2©- _wandbruntime‚1834  _runtime‚1834›ÎçÙ'ªÓO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11334632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150368596915712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚2399 % _timestamp‚1.7884183346304893e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135992543757295 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44906880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34326776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139574567269888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137584479532544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20018652˜¥¶ü)«ØO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79233654 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139574567269888 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79253692 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31353286 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2090956748800 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863814656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477108 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34326774 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34326774 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137584479532544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 \ODeviceStatsMoæáÂŵ!nitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139575241634304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863814656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20018652 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11334632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150368596915712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150368572975616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134009103580160 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853513 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚2399 % _timestamp‚1.7884183346304893e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1990744098816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44906880 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134009120619520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20018654 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11334179 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135992543757295 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699017 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139574567269888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137584497535488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165722112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2091122470912 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44906880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1990087737344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34326776 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44926916 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437748 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152459719386624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 250950656 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430ÀEÈK«2¬- _wandbruntime‚1835  _runtime‚1835x/ã&­ÝM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34340790 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 _step‚193 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137640832571392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1990907165184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139631739736576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44925334 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991563539968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152522229285376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11338803 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2399 % _timestamp‚1.7884183353056893e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1983585134991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34340792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0éáF†è&®âM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34340790 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674377728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34340790 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79266124 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137640850574336 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853513 _step‚193 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79286188 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134064024750080 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437956 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139632414114304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137640832571392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699017 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44925334 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1990907165184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚139631739736576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137640850574336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137640832571392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1990907165184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150430247653376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79286188 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44925334 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31365623 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1991563539968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79266124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152522229285376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150430223713280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863814656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674377728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44945396 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11338803 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863814656 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2399 % _timestamp‚1.7884183353056893e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11339255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134064007710720 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1983585134991 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 250950656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34340792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477316Á1¿Òá2¯- _wandbruntime‚1835  _runtime‚1835|*8W°Ñ trainer/global_step‚2399 /loss/FRt2/detection_loss‚2.7011424562869926 1loss/FBt4/assignment_loss‚0.22879050360298117 1loss/SRqqt1/detection_loss‚2.6041162527269788 1loss/SRqqt3/detection_loss‚1.7811137501682555 1loss/SRqqt4/detection_loss‚1.3358108277504261 _step‚194 0loss/FRt4/assignment_loss‚1.3303940171996753 1loss/SRqqt2/detection_loss‚2.3087231887961335 /loss/FBt4/detection_loss‚0.7029704862299175 1loss/SRqqt3/assignment_loss‚2.192293267431004 .loss/FRt1/detection_loss‚2.692089706659317 &loss/total_loss‚49.783289925453516 /loss/FRt3/assignment_loss‚3.236076407134533 .loss/FRt3/detection_loss‚2.668435134738684 % _timestamp‚1.7884183353071935e+09 0loss/FBt1/assignment_loss‚0.7618493474653102 0loss/FBt2/assignment_loss‚0.7504256976146166 0loss/FBt3/assignment_loss‚0.9685708172619343 1loss/SRqqt2/assignment_loss‚2.117372093265349 2loss/SRqqt4/assignment_loss‚1.1136783396681913 /loss/FRt2/assignment_loss‚4.064796643379407 2loss/SRqqt1/assignment_loss‚1.7921196145729885 .loss/FRt4/detection_loss‚2.061455617658794 /loss/FBt2/detection_loss‚2.9162138402462006 .loss/FBt3/detection_loss‚2.596585613489151 /loss/FRt1/assignment_loss‚4.186938139953111 .loss/FBt1/detection_loss‚2.671328162153562 epoch‚1 _runtime‚1835.044622431…>/·\±Ö 1loss/SRqqt2/assignment_loss‚2.117372093265349 2loss/SRqqt4/assignment_loss‚1.1136783396681913 /loss/FRt2/assignment_loss‚4.064796643379407 2loss/SRqqt1/assignment_loss‚1.7921196145729885 .loss/FRt4/detection_loss‚2.061455617658794 /loss/FBt2/detection_loss‚2.9162138402462006 .loss/FBt3/detection_loss‚2.596585613489151 /loss/FRt1/assignment_loss‚4.186938139953111 .loss/FBt1/detection_loss‚2.671328162153562 epoch‚1 _runtime‚1835.044622431 trainer/global_step‚2399 /loss/FRt2/detection_loss‚2.7011424562869926 1loss/FBt4/assignment_loss‚0.22879050360298117 1loss/SRqqt1/detection_loss‚2.6041162527269788 1loss/SRqqt3/detection_loss‚1.7811137501682555 1loss/SRqqt4/detection_loss‚1.3358108277504261 _step‚194 0loss/FRt4/assignment_loss‚1.3303940171996753 1loss/SRqqt2/detection_loss‚2.3087231887961335 /loss/FBt4/detection_loss‚0.7029704862299175 1loss/SRqqt3/assignment_loss‚2.192293267431004 .loss/FRt1/detection_loss‚2.692089706659317 &loss/total_loss‚49.783289925453516 /loss/FRt3/assignment_loss‚3.236076407134533 .loss/FRt3/detection_loss‚2.668435134738684 % _timestamp‚1.7884183353071935e+09 0loss/FBt1/assignment_loss‚0.7618493474653102 0loss/FBt2/assignment_loss‚0.7504256976146166 0loss/FBt3/assignment_loss‚0.9685708172619343ÂnæÝò‚(:’ ©²äÔÆóš~ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.748799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š`’€³‚(:Ý ©²äÔ³÷¬* disk./.usagePercent‚17.069100611534413% disk./.usageGB‚254.34193420410156 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚7102.02197265625 disk.sda.in‚ 126.77734375$ memory_percent‚95.61963081359863 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚4.09375 disk.nvme0n1.in‚4.09375' disk.nvme0c0n1.out‚7102.02197265625' proc.memory.availableMB‚ 717.6796875 cpu‚25.2160294365027 network.sent‚ 4.412373e+06 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0% disk.nvme0n1.out‚7102.02197265625 disk.sda3.out‚ 200.234375 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 200.234375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 0.919921875 disk.rbd0.in‚0 disk.rbd0›ÝÜÛ.out‚0 network.recv‚746945 disk.sda.out‚ 200.234375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.09375# proc.memory.rssMB‚ 2765.58984375* proc.memory.percent‚16.879820823669434u~Mœ´‚(:‘ ¸²äÔ¸…–~ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚178.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.748799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\X`Ðkµ‚(:à ¸²äÔýᨠdisk.sda1.out‚0 disk.sda3.out‚ 200.72265625 cpu‚25.24652074292422 network.recv‚754669 disk.drbd2265.out‚0 disk.dm-0.in‚4.09375 disk.drbd1716.out‚0% disk.nvme0n1.out‚7161.85009765625 disk.nvme0c0n1.in‚4.09375 disk.sda2.in‚0# proc.memory.rssMB‚ 2765.58984375 disk.sda.out‚ 200.72265625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚4.09375' disk.nvme0c0n1.out‚7161.85009765625 disk.drbd2266.in‚0 disk.rbd1.out‚ 0.919921875 disk.sda3.in‚ 126.77734375) disk./.usagePercent‚17.07076312446177% disk./.usageGB‚254.36670684814453 disk.drbd1716.in‚0# memory_percent‚95.6197738647461 network.sent‚ 4.521033e+06 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375% proc.memory.availableMB‚ 717.65625 disk.sda2.out‚0* proc.memory.percent‚16.879820823669434 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 200.72265625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0" disk.dm-0.out‚7161.85009765625 disk.rbd1.in‚0 disk.dm-1.in‚ 126.77734375z‘¶‚(:† DzäÔ¡¨ƒ~ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð¨'k·‚(:à DzäÔ disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0$ memory_percent‚95.62220573425293 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda.out‚ 201.34765625 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚7221.4208984375 disk.rbd0.out‚0 network.sent‚ 4.526454e+06$ disk./.usageGB‚254.3827362060547 disk.drbd2080.out‚0' proc.memory.availableMB‚ 717.2578125* proc.memory.percent‚16.879820823669434 disk.dm-0.in‚4.09375 disk.rbd1.in‚0* disk./.usagePercent‚17.071838868120647 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.09375 disk.sda3.in‚ 126.77734375 cpu‚25.28481559177228 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.09375 network.recv‚758456 disk.dm-1.out‚ 201.34765625 disk.rbd1.out‚ 0.919921875 disk.sda3.out‚ 201.34765625! disk.dm-0.out‚7221.4208984375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚7221.4208984375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2765.58984375K…õ¸‚(:… Ö²äÔÇ´Ž~ gpu.0.gpu‚53.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K»£g¹‚(:Ü Ö²äÔ©Í®€ cpu‚25.324492091249667 disk.rbd0.in‚0 disk.nvme0n1.in‚4.09375 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚7281.8583984375 disk.sda3.out‚ 201.890625 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375 disk.sda.out‚ 201.890625( proc.memory.availableMB‚ 717.24609375* proc.memory.percent‚16.879820823669434 network.recv‚763312 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0$ disk.nvme0n1.out‚7281.8583984375 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 201.890625 disk.rbd1.out‚ 0.919921875$ memory_percent‚95.62227725982666# proc.memory.rssMB‚ 2765.58984375 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0! disk.dm-0.out‚7281.8583984375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.09375 network.sent‚ 4.53258e+06* disk./.usagePercent‚17.072933812344594% disk./.usageGB‚254.39905166625977 disk.sda1.out‚0!¿«2º- _wandbruntime‚1896  _runtime‚1896a¸¿@*»ºT NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80757784 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2024221636096 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32134768 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚20492540 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2016770719863 _runtime‚1896.278020134 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚45619235 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚11642078 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15977 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚35122572 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2125475272192 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863814656 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24399 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137187840 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚195 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35122572 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2125588519936 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953744 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136781676275712 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚142484098320896 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32134618 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699017 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832128 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚148 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚20492542 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚155521460392960 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136781693315072 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚142483587488768 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35122574 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113247744 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80741807 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚153395871873024 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚153395847932928 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853513 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2023728806912 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80741807 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 trainer/global_step‚2417 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚150 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45635210 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2023728806912 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 _step‚195 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚35122572 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829184 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚45635210 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2125588519936 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953744 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚136781676275712 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚142484098320896 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32134618 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80757784 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699017 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832128 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚11642226 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚148 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚20492542 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚155521460392960 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚136781693315072 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚142483587488768 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚35122574 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 113247744 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914384 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80741807 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚153395871873024 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚153395847932928 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955585024 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853513 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2023728806912 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863814656 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚142484098320896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚155521323205120 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2016279805479 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80741807 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚140459858681856 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 trainer/global_step‚2417 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚150 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585024 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚45635210 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚140459858681856 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2023728806912 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 _runtime‚1897.191737662 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚138833613450087 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚35133380 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚20499044 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚80775892 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚155559564065280 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚2417 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832128 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 _RDeviceStatsMonitor.onï1\\_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _step‚196 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 % _timestamp‚1.7884183974538693e+09 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚155559564065280 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚2417 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2024093933568 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚32145914 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚45626537 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137187840 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚153433723666432 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24126 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚35133378 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955585024 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚142520782644224 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚140496688710656 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚136816987074560 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15975 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832128 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚148 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80759915 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492829184 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2125840398848 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚136816970035200 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚35133378 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚80759915 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953744 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚80775892 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 _step‚196 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2024586762752 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15977 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2125953646592 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 2017134329271 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914384 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863814656 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚138834121403831 % _timestamp‚1.7884183974538693e+09 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚142521293476352 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832128 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚153433747606528 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 _runtime‚1897.191737662 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚140496706713600 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829184 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚32145764 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2Äôgá‘À‚(:† å²äÔ ¡»~ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚187.054' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.4108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“pmÁ‚(:â å²äÔ¥…Ó€% disk./.usageGB‚254.41201782226562 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚7341.87744140625* disk./.usagePercent‚17.073803981953475' disk.nvme0c0n1.out‚7341.87744140625 disk.sda2.out‚0 disk.dm-0.in‚4.09375 disk.drbd2266.in‚0 cpu‚25.359247435544393 disk.rbd0.out‚0 disk.sda3.out‚ 202.4609375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0' proc.memory.availableMB‚ 717.4453125# proc.memory.rssMB‚ 2765.58984375 disk.nvme0c0n1.in‚4.09375 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 202.4609375$ memory_percent‚95.62106132507324 disk.drbd2266.out‚0 disk.rbd1.out‚ 0.919921875* proc.memory.percent‚16.879820823669434 network.recv‚767171 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.09375 network.sent‚ 4.538437e+06 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.dm-0.out‚7341.87744140625 disk.rbd1.in‚0 disk.sda.out‚ 202.4609375( ¥æ2Â- _wandbruntime‚1909  _runtime‚1909çñëû>Ãé§ ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37416112422943115 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5445845127105713 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 epoch‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021709250286221504 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13409961760044098 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39750605821609497 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9618846774101257 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_4_of_4‚0.005190796684473753 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5234324336051941 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16274432837963104 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5738927721977234 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18067462742328644 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09345771372318268 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1126885786652565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5222448706626892 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14660494029521942 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4444100260734558 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22284305095672607 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9553884267807007 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16541369259357452 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1460970789194107 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.26111114025115967 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23942023515701294 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3732943534851074 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15766331553459167 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11627906560897827 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09090910106897354 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014852607622742653 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17438706755638123 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9752922654151917 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42655080556869507 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9795288443565369 particle/accuracy_1_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357344746589661 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 +jet/accuracy_3_of_5‚0.10662443190813065 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 _runtime‚1909.496464634 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31592264771461487 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9713797569274902 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9219006299972534 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 *loss/val_total_loss‚52.767732290717184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17356322705745697 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9937998056411743 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9634050130844116 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_4_of_5‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13555270433425903 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05727766454219818 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1908046007156372 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11380654573440552 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5035918951034546 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 *jet/accuracy_1_of_3‚0.8338088989257812 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3862999975681305 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9422840476036072 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3107128143310547 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4809470772743225 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 _step‚197 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16742438077926636 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9874823093414307 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34233012795448303 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16486486792564392 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.95933997631073 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.015198866836726665 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3124999701976776 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8065396547317505 *jet/accuracy_1_of_4‚0.8851932883262634 particle/accuracy_3_of_4‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 +jet/accuracy_3_of_6‚0.21153846383094788 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38192734122276306 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 particle/accuracy_2_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4697226881980896 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929673075675964 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45839276909828186 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008750200271606445 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2945767045021057 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33494022488594055 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05286173149943352 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31688597798347473 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10721054673194885 particle/accuracy_1_of_1‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5246872901916504 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.028536636382341385 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.035387467592954636 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0034442066680639982 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12079869210720062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9311091303825378 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45102453231811523 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9726322889328003 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 jet/accuracy_1_of_6‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5082041025161743 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884184097565625e+09 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1577702909708023 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9831497073173523 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4806279242038727 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16491226851940155 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9880952835083008 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15453758835792542 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5681224465370178 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5724654793739319 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1255127340555191 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15872423350811005 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_5_of_7‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.38430166244506836 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16302315890789032 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09725811332464218 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11627906560897827 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5059523582458496 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15274475514888763 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20059265196323395 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.724162220954895 +particle/specificity‚0.9515019059181213 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18611110746860504 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022761298343539238 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purå£ôBõTity‚0.3996172249317169 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9047619700431824 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3056652843952179 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008918138220906258 *jet/accuracy_2_of_6‚0.3974359333515167 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5381630659103394 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.597569465637207 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4837161600589752 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.529967188835144 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42193636298179626 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1303897500038147 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.898148238658905 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15997780859470367 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_1_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3779693841934204 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46092984080314636 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45839276909828186 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929673075675964 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4458678066730499 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.504061758518219 +jet/accuracy_2_of_3‚0.22425466775894165 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.152162566781044 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27015969157218933 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1598881334066391 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9681840538978577 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9554924964904785 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4781636595726013 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 5 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.98759925365448 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014506989158689976 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23978613317012787 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4568180441856384 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14318181574344635 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 particle/accuracy_6_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024769585579633713 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10890337079763412 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 )jet/accuracy_2_of_5‚0.438972145318985 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9682093262672424 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4290013611316681 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42953506112098694 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37527230381965637 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3114583492279053 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28538551926612854 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07028363645076752 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6722680330276489 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32340067625045776 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49221256375312805 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,particle/sensitivity‚0.26387739181518555 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9714592099189758 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1949746012687683 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31201550364494324 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43903258442878723 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6080586314201355 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_6‚1 jet/accuracy_3_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18105530738830566 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2258537858724594 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4371649920940399 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_6_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9878625869750977 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05000000074505806 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12551440298557281 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_3_of_4‚0.057176996022462845 particle/accuracy_5_of_5‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.902630627155304 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2361110895872116 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.463733434677124 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_5_of_7‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.8958333730697632 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.950634777545929 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11797469109296799 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4962938129901886 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12007945775985718 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3030303120613098 6validation_average_jet_accuracy‚0.4100385904312134 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021182918921113014 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4698604643344879 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14220792055130005 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4739712178707123 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0957593098282814 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19864849746227264 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9831737875938416 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4933798611164093 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_7_of_7‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5127838850021362 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0520833320915699 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8441377282142639 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6346542239189148 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9373264312744141 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05291444808244705 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5090513825416565 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31284722685813904 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25808873772621155 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11546705663204193 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9644746780395508 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 'particle/f_score‚0.3456137478351593 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4220947325229645 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8833333849906921 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44013655185699463 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12613941729068756 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.45216643810272217 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9477183222770691 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5360118746757507 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚2417 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3490448594093323 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47790658473968506 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04330601170659065 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9846051335334778 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0342261902987957 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9473869204521179 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13628116250038147 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36585041880607605 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9782612919807434 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.323723703622818 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9867871999740601 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35037222504615784 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 particle/accuracy_3_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8936170339584351 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22541548311710358 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9672413468360901 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4120370149612427 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38180825114250183 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3353758156299591 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44141265749931335 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9651833772659302 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9884479641914368 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.302198201417923 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9959776997566223 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9686500430107117 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666269302368 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.33565372228622437 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 jet/accuracy_7_of_7‚0)-Óý*Äî§ ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.950634777545929 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11797469109296799 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4962938129901886 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12007945775985718 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3030303120613098 6validation_average_jet_accuracy‚0.4100385904312134 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021182918921113014 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4698604643344879 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14220792055130005 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4739712178707123 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.46408048272132874 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.0957593098282814 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19864849746227264 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9831737875938416 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4933798611164093 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_7_of_7‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5127838850021362 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0520833320915699 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8441377282142639 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6346542239189148 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9373264312744141 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05291444808244705 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5090513825416565 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31284722685813904 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25808873772621155 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11546705663204193 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_4‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9644746780395508 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 'particle/f_score‚0.3456137478351593 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4220947325229645 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.8833333849906921 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44013655185699463 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12613941729068756 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.45216643810272217 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9477183222770691 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5360118746757507 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 trainer/global_step‚2417 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3490448594093323 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.47790658473968506 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04330601170659065 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9846051335334778 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0342261902987957 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9473869204521179 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13628116250038147 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.36585041880607605 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9782612919807434 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.323723703622818 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9867871999740601 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt1SRqqt1FBt/FBt_purity‚0.9940475821495056 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.35037222504615784 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 particle/accuracy_3_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.8936170339584351 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22541548311710358 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9672413468360901 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4120370149612427 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.38180825114250183 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3353758156299591 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44141265749931335 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9651833772659302 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9884479641914368 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.302198201417923 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9959776997566223 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9686500430107117 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9904166460037231 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666269302368 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.33565372228622437 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.37416112422943115 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3235294222831726 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5445845127105713 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 epoch‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021709250286221504 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13409961760044098 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.7916666865348816 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.3975S&±õøh0605821609497 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9618846774101257 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,jet/accuracy_4_of_4‚0.005190796684473753 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5234324336051941 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16274432837963104 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5738927721977234 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18067462742328644 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09345771372318268 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1126885786652565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5222448706626892 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.14660494029521942 jet/accuracy_5_of_6‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4444100260734558 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22284305095672607 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9553884267807007 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16541369259357452 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1460970789194107 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.26111114025115967 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23942023515701294 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3732943534851074 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8333333134651184 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15766331553459167 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11627906560897827 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09090910106897354 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014852607622742653 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17438706755638123 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9752922654151917 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42655080556869507 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9795288443565369 particle/accuracy_1_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357344746589661 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 +jet/accuracy_3_of_5‚0.10662443190813065 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 _runtime‚1909.496464634 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.31592264771461487 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9713797569274902 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9219006299972534 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 *loss/val_total_loss‚52.767732290717184 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17356322705745697 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9937998056411743 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9634050130844116 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_4_of_5‚1 ) Purity/0FRt0SRqqt4FBt/FBt_purity‚0.75 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13555270433425903 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05727766454219818 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1908046007156372 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11380654573440552 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5035918951034546 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0006613756995648146 *jet/accuracy_1_of_3‚0.8338088989257812 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3862999975681305 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9422840476036072 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3107128143310547 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4809470772743225 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 _step‚197 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16742438077926636 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9874823093414307 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34233012795448303 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16486486792564392 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.95933997631073 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.015198866836726665 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3124999701976776 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3866666853427887 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8065396547317505 *jet/accuracy_1_of_4‚0.8851932883262634 particle/accuracy_3_of_4‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 +jet/accuracy_3_of_6‚0.21153846383094788 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38192734122276306 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9735848903656006 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 particle/accuracy_2_of_6‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4697226881980896 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9929673075675964 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.45839276909828186 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.008750200271606445 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2945767045021057 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33494022488594055 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05286173149943352 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.31688597798347473 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10721054673194885 particle/accuracy_1_of_1‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5246872901916504 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.028536636382341385 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.035387467592954636 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0034442066680639982 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12079869210720062 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9311091303825378 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45102453231811523 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9726322889328003 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 jet/accuracy_1_of_6‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5082041025161743 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884184097565625e+09 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1577702909708023 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9831497073173523 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4806279242038727 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.16491226851940155 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9880952835083008 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15453758835792542 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5681224465370178 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5724654793739319 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1255127340555191 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15872423350811005 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_5_of_7‚1 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.38430166244506836 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16302315890789032 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09725811332464218 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11627906560897827 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5059523582458496 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15274475514888763 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20059265196323395 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.724162220954895 +particle/specificity‚0.9515019059181213 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.18611110746860504 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.022761298343539238 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.07999999821186066 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3996172249317169 -jet/accuracy_4_of_5‚0.0026881720405071974 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9047619700431824 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3056652843952179 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.008918138220906258 *jet/accuracy_2_of_6‚0.3974359333515167 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5381630659103394 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.597569465637207 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4837161600589752 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.529967188835144 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.42193636298179626 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1303897500038147 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.898148238658905 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15997780859470367 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_1_of_2‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3779693841934204 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46092984080314636 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.45839276909828186 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9929673075675964 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4458678066730499 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.504061758518219 +jet/accuracy_2_of_3‚0.22425466775894165 6 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.800000011920929 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.152162566781044 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.27015969157218933 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1598881334066391 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9681840538978577 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9554924964904785 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4781636595726013 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 5 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.98759925365448 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014506989158689976 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13829787075519562 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23978613317012787 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4568180441856384 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14318181574344635 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 particle/accuracy_6_of_6‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.024769585579633713 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0013227513991296291 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10890337079763412 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 )jet/accuracy_2_of_5‚0.438972145318985 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5701754093170166 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9682093262672424 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4290013611316681 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42953506112098694 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.37527230381965637 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3114583492279053 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28538551926612854 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07028363645076752 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6722680330276489 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.32340067625045776 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.49221256375312805 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,particle/sensitivity‚0.26387739181518555 ) Purity/0FRt*SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9714592099189758 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1949746012687683 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.31201550364494324 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43903258442878723 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6080586314201355 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_6‚1 jet/accuracy_3_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18105530738830566 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8728070259094238 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2258537858724594 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4371649920940399 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_6_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9878625869750977 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05000000074505806 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.12551440298557281 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,jet/accuracy_3_of_4‚0.057176996022462845 particle/accuracy_5_of_5‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.902630627155304 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2361110895872116 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.463733434677124 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_5_of_7‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.8958333730697632 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0Å}j¦F’Å‚(:‡ ô²äÔ㧩~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚122.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kp{fÆ‚(:Û ô²äÔÁÂÒ€ disk.rbd1.out‚ 0.919921875 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.09375 disk.dm-0.out‚7869.080078125 disk.nvme0n1.in‚4.09375* proc.memory.percent‚16.916370391845703 network.sent‚ 4.646949e+06 network.recv‚774740 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0& proc.memory.availableMB‚ 951.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 202.83203125 disk.drbd1716.out‚0 disk.sda3.out‚ 202.83203125! proc.memory.rssMB‚ 2771.578125 disk.sda1.out‚0# disk.nvme0n1.out‚7869.080078125 disk.nvme0c0n1.in‚4.09375 disk.sda2.in‚0% disk./.usageGB‚254.87129592895508 disk.dm-1.out‚ 202.83203125 disk.rbd1.in‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚7869.080078125 cpu‚25.36611306636017 proc.cpu.threads‚17* disk./.usagePercent‚17.104626521053405 disk.sda2.out‚0$ memory_percent‚94.19145584106445Ž q‘Ç‚(:† ƒ³äÔå´~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚152.555' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.511 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YrV!aÈ‚(:Ö ƒ³äÔþÞ´€$ memory_percent‚94.19143199920654 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 203.578125 network.sent‚ 4.770541e+06 disk.sda.out‚ 203.578125 disk.nvme0c0n1.in‚4.09375% disk.nvme0c0n1.out‚7933.095703125 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚4.09375 disk.drbd2266.in‚0 network.recv‚781579 disk.dm-0.out‚7933.095703125 disk.rbd1.out‚ 0.919921875( proc.memory.availableMB‚ 951.67578125( disk./.usagePercent‚17.1055939735253% disk./.usageGB‚254.88571166992188* proc.memory.percent‚16.916370391845703 cpu‚25.363701504560467 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0 disk.nvme0n1.in‚4.09375# disk.nvme0n1.out‚7933.095703125! proc.memory.rssMB‚ 2771.578125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 203.578125 disk.drbd2265.out‚0¨£2É- _wandbruntime‚1937  _runtime‚1937ß ø•ÉÊà trainer/global_step‚2449 % _timestamp‚1.7884184373360996e+09 _runtime‚1937.073607731 _step‚198 ' lr-AdamW/pg1‚0.00016333871869910545 ' lr-AdamW/pg2‚0.00016333871869910545ÒªÎËÈ ' lr-AdamW/pg1‚0.00016333871869910545 ' lr-AdamW/pg2‚0.00016333871869910545 trainer/global_step‚2449 % _timestamp‚1.7884184373360996e+09 _runtime‚1937.073607731 _step‚198Æž|2Ì- _wandbruntime‚1937  _runtime‚1937ÃË 4ÝÍÓO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚2449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2157254953984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884184373369508e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2054749872640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46294890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35708354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2054749872640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on„ÍÌYü_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138977816955904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20836804 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142722441292288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46314927 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155837281069568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35708352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82003242 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142722441292288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141025016972525 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35708352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20836806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11844776 _runtime‚1937.074819017“ðˆÞ'ÎØO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142722423289344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157994536023552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35708352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82003242 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142722441292288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141025016972525 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2054093510656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35708352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144777191164928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142722423289344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35708354 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20836806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11844776 _runtime‚1937.074819017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚2449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32681580 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2157254953984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11845200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884184373369508e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46314927 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2054749872640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2157424869888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46294890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35708354 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2054749872640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144777191164928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144776516800000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138977816955904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20836804 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142722441292288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193856000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144776516800000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46314927 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2054093510656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155837281069568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2046528539509 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46294890 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82023281 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144834363644928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20844974 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32694642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Å^ô›ïÑáM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35722370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144834363644928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144834363644928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20844974 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82055777 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155898955747328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82035712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32694211 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2449 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2055569313792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144833689266688 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139032721086464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚200 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32694642 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158057038885376 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35722370 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139032738125824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2047344974473 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46333407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2054912938496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142778794331136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46313344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2055569313792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11849239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÈKiÐ2Ò- _wandbruntime‚1937  _runtime‚1937ìؾúYÓÓ 1loss/SRqqt3/assignment_loss‚2.145024094809877 /loss/FBt3/assignment_loss‚0.863619090244174 .loss/FBt3/detection_loss‚2.514010715484619 .loss/FRt1/assignment_loss‚4.46702624530327 2loss/SRqqt2/assignment_loss‚2.0232882322195698 /loss/FRt1/detection_loss‚2.2548825944342266 1loss/SRqqt4/assignment_loss‚1.416301729157567 .loss/FRt3/detection_loss‚2.380057764296629 /loss/FRt4/detection_loss‚1.8471061097314725 trainer/global_step‚2449 0loss/FRt3/assignment_loss‚3.7849937677383423 0loss/FBt1/assignment_loss‚0.5494418218731877 /loss/FBt4/detection_loss‚0.7301332927846278 % _timestamp‚1.7884184380372746e+09 0loss/FRt2/assignment_loss‚3.7656569389196544 0loss/FRt4/assignment_loss‚1.9915351970837667 2loss/SRqqt1/assignment_loss‚2.0567669404877558 1loss/SRqqt1/detection_loss‚2.1652211596568427 /loss/FBt2/detection_loss‚3.9205366373062134 1loss/FBt2/assignment_loss‚0.34911378969748813 1loss/SRqqt3/detection_loss‚2.0634514599087392 1loss/SRqqt4/detection_loss‚1.3132821161092982 /loss/FBt1/detection_loss‚1.1457798234094945 epoch‚2 _runtime‚ 1937.77468643 1loss/FBt4/assignment_loss‚0.23845156704052564 .loss/FRt2/detection_loss‚3.252529854957874 1loss/SRqqt2/detection_loss‚2.4993415860568775 &loss/total_loss‚49.737552528712094 _step‚201σµ$^ÔØ 1loss/SRqqt4/assignment_loss‚1.416301729157567 .loss/FRt3/detection_loss‚2.380057764296629 /loss/FRt4/detection_loss‚1.8471061097314725 trainer/global_step‚2449 0loss/FRt3/assignment_loss‚3.7849937677383423 0loss/FBt1/assignment_loss‚0.5494418218731877 /loss/FBt4/detection_loss‚0.7301332927846278 % _timestamp‚1.7884184380372746e+09 0loss/FRt2/assignment_loss‚3.7656569389196544 0loss/FRt4/assignment_loss‚1.9915351970837667 2loss/SRqqt1/assignment_loss‚2.0567669404877558 1loss/SRqqt1/detection_loss‚2.1652211596568427 /loss/FBt2/detection_loss‚3.9205366373062134 1loss/FBt2/assignment_loss‚0.34911378969748813 1loss/SRqqt3/detection_loss‚2.0634514599087392 1loss/SRqqt4/detection_loss‚1.3132821161092982 /loss/FBt1/detection_loss‚1.1457798234094945 epoch‚2 _runtime‚ 1937.77468643 1loss/FBt4/assignment_loss‚0.23845156704052564 .loss/FRt2/detection_loss‚3.252529854957874 1loss/SRqqt2/detection_loss‚2.4993415860568775 &loss/total_loss‚49.737552528712094 _step‚201 1loss/SRqqt3/assignment_loss‚2.145024094809877 /loss/FBt3/assignment_loss‚0.863619090244174 .loss/FBt3/detection_loss‚2.514010715484619 .loss/FRt1/assignment_loss‚4.46702624530327 2loss/SRqqt2/assignment_loss‚2.0232882322195698 /loss/FRt1/detection_loss‚2.2548825944342266Éˤ—»‘Õ‚(:† ’³äÔý½·~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x&‰aÖ‚(:Ö ’³äÔý¡Ê€ network.recv‚785898 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2771.578125* disk./.usagePercent‚17.107261606603338% disk./.usageGB‚254.91056060791016 disk.rbd1.in‚0$ memory_percent‚94.19341087341309 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚4.3125 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.36118936123888 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.3125# disk.nvme0n1.out‚8452.537109375 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚8452.537109375% disk.nvme0c0n1.out‚8452.537109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 204.0546875 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 204.0546875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 951.3515625 disk.dm-1.out‚ 204.0546875 disk.rbd1.out‚ 0.919921875 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.3125* proc.memory.percent‚16.916370391845703 network.sent‚ 4.776526e+06¨gí’ׂ(:‡ ¡³äÔÊäý} gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µíƒñgØ‚(:Ü ¡³äÔîÏÔ€ disk.drbd1716.in‚0' disk.nvme0c0n1.out‚8511.03857421875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 204.80078125 disk.drbd2266.out‚0 disk.sda.out‚ 204.80078125* proc.memory.percent‚16.916370391845703 network.sent‚ 4.885082e+06% disk.nvme0n1.out‚8511.03857421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 cpu‚25.35896863098828 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.3125 disk.dm-0.in‚4.3125 disk.rbd1.out‚ 0.919921875( proc.memory.availableMB‚ 951.58984375! proc.memory.rssMB‚ 2771.578125 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 204.80078125" disk.dm-0.out‚8511.03857421875$ memory_percent‚94.19195652008057 network.recv‚792832! disk./.usageGB‚254.9365234375 disk.nvme0n1.in‚4.3125 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0* disk./.usagePercent‚17.109003993881377€¿2Ù- _wandbruntime‚1973  _runtime‚1973~sàLÉÚà ' lr-AdamW/pg1‚0.00016290383747609202 ' lr-AdamW/pg2‚0.00016290383747609202 trainer/global_step‚2499 % _timestamp‚1.7884184740996006e+09 _runtime‚1973.837178132 _step‚202bÄœÎÛÈ trainer/global_step‚2499 % _timestamp‚1.7884184740996006e+09 _runtime‚1973.837178132 _step‚202 ' lr-AdamW/pg1‚0.00016290383747609202 ' lr-AdamW/pg2‚0.00016290383747609202Ê&©Û2Ü- _wandbruntime‚1973  _runtime‚1973¶×™Ø'ÝÒO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145540075231744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2095735678464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83650133 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 _runtime‚1973.838160405 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141723023483904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12070986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚203 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36409152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36409154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21245204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788418474100364e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2088018316669|HaÝ'Þ×O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚147635154548224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158919841892864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2095735678464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2095735678464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83650133 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚141723040523264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2087363878917 _runtime‚1973.838160405 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161120140085760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141723023483904 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47220942 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12070986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2095079316480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2200132471296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚203 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36409152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚147635828913152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145540093234688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2095079316480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145540075231744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36409154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200298192896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83630094 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21245204 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12070532 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788418474100364e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2088018316669 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145540075231744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚147635828913152ËXbî2ß- _wandbruntime‚1974  _runtime‚1974Ê÷âàÝM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33328956 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XIDeviceStat ¥Ù‹%sMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2096555119616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36423170 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21253374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47239396 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21253372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47259459 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 % _timestamp‚1.7884184748088093e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152 _runtime‚1974.546814775 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145596446273536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12075124 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2096555119616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2088834751841 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12075582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143866779405665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚204 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141777927614464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚2499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36423168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158981492630528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2201154613760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚204 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2200984711168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141777927614464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143866107928345 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145596428270592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚2499 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36423168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36423168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158981492630528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2201154613760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33328496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21253374 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83682629 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47239396 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83682629 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2088180313881 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21253372 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47259459 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2095898744320 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83662564 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145596446273536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47259459 % _timestamp‚1.7884184748088093e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚147693001393152Ìôíâ2â- _wandbruntime‚1974  _runtime‚1974eVãÐ /loss/FRt2/assignment_loss‚3.251296881834666 .loss/FBt1/detection_loss‚9.842114649713345 /loss/FBt4/detection_loss‚1.0029934580476645 trainer/global_step‚2499 _runtime‚1974.548377324 2loss/SRqqt1/assignment_loss‚2.1800952066595736 .loss/FRt3/detection_loss‚2.465003875168887 .loss/FBt3/detection_loss‚3.153285101056099 % _timestamp‚1.7884184748109229e+09 2loss/SRqqt2/assignment_loss‚1.9524387474706832 2loss/SRqqt3/assignment_loss‚1.7296556855847196 2loss/SRqqt4/assignment_loss‚1.1611153390506903 .loss/FRt4/detection_loss‚2.533947580917315 1loss/SRqqt3/detection_loss‚2.1934575838286703 1loss/SRqqt2/detection_loss‚1.9310589477102806 1loss/SRqqt4/detection_loss‚1.2640807292362055 _step‚205 .loss/FRt1/detection_loss‚2.793664786964655 /loss/FRt2/detection_loss‚2.3343444373872546 /loss/FRt4/assignment_loss‚2.316500109705058 1loss/SRqqt1/detection_loss‚2.6584836210195837 %loss/total_loss‚58.79748246433922 epoch‚2 /loss/FRt3/assignment_loss‚4.002197460694746 .loss/FBt2/detection_loss‚3.176268148422241 0loss/FBt1/assignment_loss‚0.7220173901042699 0loss/FBt2/assignment_loss‚0.6584410220384598 1loss/FBt4/assignment_loss‚0.26817231444479067 /loss/FRt1/assignment_loss‚4.711411226540804 0loss/FBt3/assignment_loss‚0.4954381607385585#Ù«E[äÕ /loss/FRt2/assignment_loss‚3.251296881834666 .loss/FBt1/detection_loss‚9.842114649713345 /loss/FBt4/detection_loss‚1.0029934580476645 trainer/global_step‚2499 _runtime‚1974.548377324 2loss/SRqqt1/assignment_loss‚2.1800952066595736 .loss/FRt3/detection_loss‚2.465003875168887 .loss/FBt3/detection_loss‚3.153285101056099 % _timestamp‚1.7884184748109229e+09 2loss/SRqqt2/assignment_loss‚1.9524387474706832 2loss/SRqqt3/assignment_loss‚1.7296556855847196 2loss/SRqqt4/assignment_loss‚1.1611153390506903 .loss/FRt4/detection_loss‚2.533947580917315 1loss/SRqqt3/detection_loss‚2.1934575838286703 1loss/SRqqt2/detection_loss‚1.9310589477102806 1loss/SRqqt4/detection_loss‚1.2640807292362055 _step‚205 .loss/FRt1/detection_loss‚2.793664786964655 /loss/FRt2/detection_loss‚2.3343444373872546 /loss/FRt4/assignment_loss‚2.316500109705058 1loss/SRqqt1/detection_loss‚2.6584836210195837 %loss/total_loss‚58.79748246433922 epoch‚2 /loss/FRt3/assignment_loss‚4.002197460694746 .loss/FBt2/detection_loss‚3.176268148422241 0loss/FBt1/assignment_loss‚0.7220173901042699 0loss/FBt2/assignment_loss‚0.6584410220384598 1loss/FBt4/assignment_loss‚0.26817231444479067 /loss/FRt1/assignment_loss‚4.711411226540804 0loss/FBt3/assignment_loss‚0.4954381607385585ÍYðÖ“’å‚(:‡ °³äÔ§˜£~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X½Uæ‚(:Ê °³äÔ»œ˜ disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.916370391845703 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17" disk.dm-0.out‚8560.15966796875 disk.drbd2266.out‚0 disk.sda.out‚205.25 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚205.25 disk.nvme0n1.in‚4.3125% disk.nvme0n1.out‚8560.15966796875! proc.memory.rssMB‚ 2771.578125* disk./.usagePercent‚17.110363393888196% disk./.usageGB‚254.95677947998047 disk.sda.in‚ 126.77734375 disk.sda1.out‚0' disk.nvme0c0n1.out‚8560.15966796875 disk.sda3.out‚205.25 cpu‚25.35652002991511 disk.rbd1.out‚ 0.919921875 network.sent‚ 4.891404e+06 disk.dm-0.in‚4.3125 disk.drbd1716.in‚0$ proc.memory.availableMB‚951.0625 disk.nvme0c0n1.in‚4.3125 disk.drbd2265.out‚0$ memory_percent‚94.19517517089844 network.recv‚797641ä-¸Š›ç‚(: ¿³äÔ¹ÿÔ~ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.0' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.199999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ða2cè‚(:Ø ¿³äÔ³é¨ disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 network.recv‚804142 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 205.8203125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.3125 disk.sda3.out‚ 205.8203125 disk.dm-0.in‚4.3125 disk.rbd1.out‚ 0.919921875 disk.nvme0n1.in‚4.3125 disk.sda2.in‚0 cpu‚25.354365503334055 disk.dm-1.out‚ 205.8203125 disk.sda2.out‚0# memory_percent‚94.1936731338501( proc.memory.availableMB‚ 951.30859375* proc.memory.percent‚16.916370391845703 disk.rbd1.in‚0& disk.nvme0c0n1.out‚8627.3017578125! proc.memory.rssMB‚ 2771.578125 disk.sda1.in‚0$ disk.nvme0n1.out‚8627.3017578125 disk.drbd2265.out‚0 network.sent‚ 4.999641e+06 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0) disk./.usagePercent‚17.11124713189639$ disk./.usageGB‚254.9699478149414! disk.dm-0.out‚8627.3017578125 disk.drbd2266.in‚0 disk.rbd0.in‚0/“î”é‚(:’ γäÔ–¼œ~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.656' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.731200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œbøtê‚(:é γäÔž¶€ disk.sda3.in‚ 126.77734375 network.recv‚808659 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.005859375 disk.drbd1716.out‚0$ disk.nvme0n1.out‚8675.8408203125 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.31640625 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.31640625* proc.memory.percent‚16.916370391845703 proc.cpu.threads‚17* disk./.usagePercent‚17.112262457777188 disk.sda.out‚ 206.63671875 disk.sda2.in‚0 disk.drbd2265.in‚0! disk.dm-0.out‚8675.8408203125 cpu‚25.352146245255224 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚8675.8408203125 disk.sda3.out‚ 206.63671875 disk.drbd2080.in‚0# memory_percent‚94.1936731338501 network.sent‚ 5.005672e+06 disk.rbd0.in‚0! proc.memory.rssMB‚ 2771.578125 disk.dm-1.out‚ 206.63671875 disk.sda2.out‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 951.30859375% disk./.usageGB‚254.98507690429688 disk.nvme0n1.in‚ 4.31640625]ß?ò2ë- _wandbruntime‚2011  _runtime‚2011XýÈóÉìà % _timestamp‚1.7884185113488767e+09 _runtime‚2011.086397437 _step‚206 ' lr-AdamW/pg1‚0.00016245258435552444 ' lr-AdamW/pg2‚0.00016245258435552444 trainer/global_step‚2549[­ÎíÈ _runtime‚2011.086397437 _step‚206 ' lr-AdamW/pg1‚0.00016245258435552444 ' lr-AdamW/pg2‚0.00016245258435552444 trainer/global_step‚2549 % _timestamp‚1.7884185113488767e+09ÎÁF”-2î- _wandbruntime‚2011  _runtime‚2011ÂW˜-ÓïÓO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚207 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2011.087281365 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 % _timestamp‚1.7884185113495479e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994ªÄ-/Þ'ðØO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚207 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37109952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85276985 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150493792296448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12296551 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚2011.087281365 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37109954 % _timestamp‚1.7884185113495479e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37109954 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2243316066304 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48146994 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150494466661376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2136721484288 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844ÏY6¦2ñ- _wandbruntime‚2011  _runtime‚2011n¡ â&òÜM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2136884550144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2244184223232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚2549 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2137540925440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2137540925440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _runtime‚2011.802122692 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164308213736960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37123970 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162064005573632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146652149795753 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144523134142464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 _step‚208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148414098215936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚148414080212992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146652149795753 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162064029513728 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48185511 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150551639141376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144523134142464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85289416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185454080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150550964763136 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85289416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21661774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12300560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144523151181824 _step‚208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33962332 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2244022709248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37123968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚148414098215936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85309481 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146652821273073 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚148414080212992 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48185511 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85309481 T@Deviê¸[— ceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150551639141376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2129015653289 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _runtime‚2011.802122692 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2129670091249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164308213736960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240Ðů Ñ2ô- _wandbruntime‚2011  _runtime‚2011K²Í[õÕ 0loss/FBt3/assignment_loss‚0.7441316209733486 /loss/FRt2/detection_loss‚2.7191420570015907 %loss/total_loss‚50.92355545564796 .loss/FRt3/assignment_loss‚3.19086125420361 2loss/SRqqt2/assignment_loss‚2.3974923531751373 2loss/SRqqt3/assignment_loss‚1.7017928458572853 1loss/FBt4/assignment_loss‚0.18402003627925428 /loss/FRt3/detection_loss‚2.5160332343927245 _step‚209 /loss/FRt1/assignment_loss‚4.272340268623538 2loss/SRqqt1/assignment_loss‚2.0091217453591526 /loss/FBt3/detection_loss‚3.3095403015613556 1loss/FBt1/assignment_loss‚0.46129009846263214 0loss/FBt2/assignment_loss‚0.8160550333559513 1loss/SRqqt1/detection_loss‚2.5062226466834545 1loss/SRqqt4/detection_loss‚1.2473174748413356 /loss/FBt4/detection_loss‚0.7548483828883199 % _timestamp‚1.7884185120660563e+09 0loss/FRt2/assignment_loss‚3.5277289506047964 1loss/SRqqt4/assignment_loss‚1.248980184695051 /loss/FRt4/detection_loss‚1.7714054340556744 /loss/FBt1/detection_loss‚3.3952045400937396 epoch‚2 0loss/FRt4/assignment_loss‚1.8609896061881896 1loss/SRqqt2/detection_loss‚2.4869414970681474 trainer/global_step‚2549 .loss/FRt1/detection_loss‚2.353041259253897 0loss/SRqqt3/detection_loss‚2.219939009774299 /loss/FBt2/detection_loss‚3.2291156202554703 _runtime‚2011.803495763£ð¬Š`öÚ 0loss/FRt4/assignment_loss‚1.8609896061881896 1loss/SRqqt2/detection_loss‚2.4869414970681474 trainer/global_step‚2549 .loss/FRt1/detection_loss‚2.353041259253897 0loss/SRqqt3/detection_loss‚2.219939009774299 /loss/FBt2/detection_loss‚3.2291156202554703 _runtime‚2011.803495763 0loss/FBt3/assignment_loss‚0.7441316209733486 /loss/FRt2/detection_loss‚2.7191420570015907 %loss/total_loss‚50.92355545564796 .loss/FRt3/assignment_loss‚3.19086125420361 2loss/SRqqt2/assignment_loss‚2.3974923531751373 2loss/SRqqt3/assignment_loss‚1.7017928458572853 1loss/FBt4/assignment_loss‚0.18402003627925428 /loss/FRt3/detection_loss‚2.5160332343927245 _step‚209 /loss/FRt1/assignment_loss‚4.272340268623538 2loss/SRqqt1/assignment_loss‚2.0091217453591526 /loss/FBt3/detection_loss‚3.3095403015613556 1loss/FBt1/assignment_loss‚0.46129009846263214 0loss/FBt2/assignment_loss‚0.8160550333559513 1loss/SRqqt1/detection_loss‚2.5062226466834545 1loss/SRqqt4/detection_loss‚1.2473174748413356 /loss/FBt4/detection_loss‚0.7548483828883199 % _timestamp‚1.7884185120660563e+09 0loss/FRt2/assignment_loss‚3.5277289506047964 1loss/SRqqt4/assignment_loss‚1.248980184695051 /loss/FRt4/detection_loss‚1.7714054340556744 /loss/FBt1/detection_loss‚3.3952045400937396 epoch‚2Ñå£Í…’÷‚(:‡ ݳäÔÁÁŠ~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.768' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9536 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mݪ‚lø‚(:á ݳäÔÙÓ£€ disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0* proc.memory.percent‚16.916370391845703 cpu‚25.3499403729041 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125 disk.sda.out‚ 207.21875 disk.nvme0c0n1.in‚ 4.31640625 disk.dm-0.in‚ 4.31640625 disk.dm-1.out‚ 207.21875 disk.drbd2266.out‚0 network.sent‚ 5.011488e+06 disk.drbd1716.out‚0% disk.nvme0n1.out‚8727.84912109375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375( proc.memory.availableMB‚ 951.48828125 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.31640625 disk.sda3.out‚ 207.21875" disk.dm-0.out‚8727.84912109375 proc.cpu.threads‚17) disk./.usagePercent‚17.11338069868675% disk./.usageGB‚255.00173950195312 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚94.19257640838623 network.recv‚812488 disk.rbd1.out‚ 1.068359375' disk.nvme0c0n1.out‚8727.84912109375½ ƒ’ù‚(:‡ ì³äÔ„ý} gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ojï¹mú‚(:â ì³äÔÿ˜Š€% disk.nvme0n1.out‚8775.47900390625 disk.nvme0c0n1.in‚ 4.31640625 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚820390) disk./.usagePercent‚17.11194500843473 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125 disk.dm-1.out‚ 207.78125 network.sent‚ 5.120487e+06 disk.drbd1716.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚8775.47900390625( proc.memory.availableMB‚ 951.24609375$ disk./.usageGB‚254.9803466796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 cpu‚25.347767872186207 disk.sda1.out‚0 disk.sda3.out‚ 207.78125 disk.dm-1.in‚ 126.77734375$ memory_percent‚94.19405460357666 disk.sda.out‚ 207.78125 disk.nvme0n1.in‚ 4.31640625 disk.rbd1.out‚ 1.068359375* proc.memory.percent‚16.916370391845703 disk.sda.in‚ 126.77734375' disk.nvme0c0n1.out‚8775.47900390625 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.31640625 disk.sda1.in‚0À.Ô 2û- _wandbruntime‚2049  _runtime‚2049g ²¦Èü ' lr-AdamW/pg1‚0.00016198505337787673 ' lr-AdamW/pg2‚0.00016198505337787673 trainer/global_step‚2599 $ _timestamp‚1.788418549388423e+09 _runtime‚2049.125957518 _step‚210×3UŸÍýÇ $ _timestamp‚1.788418549388423e+09 _runtime‚2049.125957518 _step‚210 ' lr-AdamW/pg1‚0.00016198505337787673 ' lr-AdamW/pg2‚0.00016198505337787673 trainer/global_step‚2599Ò\·Ö2þ- _wandbruntime‚2049  _runtime‚2049¯6TàÙ'ÿÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2169034557733 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2177050928128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34584095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884185493892932e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151175397119488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37810752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149383142574749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151175397119488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚211 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49093083 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34584514 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147213453579264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2049.127097239 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37810752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153353104409600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167370970259456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508gÞ'€ØO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153352430044672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37810752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153353104409600 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86903837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2286078540288 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149383142574749 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151175397119488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚2599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86883798 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147213436539904 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _step‚211 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49093083 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37810754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34584514 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165084915659264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147213453579264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2049.127097239 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37810752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49073046 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153353104409600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12522091 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167371157824000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151175379116544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167370970259456 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12522508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚417 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165084891719168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2169034557733 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2177050928128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34584095 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884185493892932e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚153352430044672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151175397119488 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884185500968843e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151231732155392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37824770 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_bat ·Ä!ch_end/active.all.peak‚24400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49111563 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49091500 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22070174 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86936333 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37824768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚153409602511360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2049.835104445 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37824768 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2170505430657 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2049.835104445 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2178526731264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167433483933696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149438863140481 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37824768 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86916268 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34596711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2169850992697 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _step‚212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37824770 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884185500968843e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151231732155392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149438191663161 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153410276889600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37824770 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2286941476864 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147268357709824 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165146566396928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86936333 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚418 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151231750158336 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚420 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12526957 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844Ôw5ò2„- _wandbruntime‚2049  _runtime‚2049Õ£À½Q…Ë 2loss/SRqqt4/assignment_loss‚1.0686847364382916 .loss/FBt1/detection_loss‚2.676278779655695 .loss/FBt3/detection_loss‚4.074291666348775 $loss/total_loss‚49.7148003513511 2loss/SRqqt1/assignment_loss‚1.8265606065190607 /loss/FRt1/detection_loss‚2.2544817298650743 epoch‚2 _runtime‚2049.836489739 0loss/FBt4/assignment_loss‚0.3037115617646267 .loss/FRt2/detection_loss‚2.706211948033535 0loss/SRqqt2/detection_loss‚2.224140871937076 1loss/SRqqt2/assignment_loss‚1.473756682088909 /loss/FRt4/detection_loss‚2.0043113227002323 0loss/SRqqt3/detection_loss‚1.786235196410485 .loss/FBt2/detection_loss‚3.232087880373001 trainer/global_step‚2599 /loss/FRt3/assignment_loss‚3.888651539297665 /loss/FRt4/assignment_loss‚2.036660080154737 0loss/FBt1/assignment_loss‚0.5572146886786526 0loss/FBt2/assignment_loss‚0.6194863064331002 .loss/FBt3/assignment_loss‚0.55522770992593 1loss/SRqqt1/detection_loss‚2.6869546953708894 1loss/SRqqt4/detection_loss‚1.0536233929329013 /loss/FBt4/detection_loss‚0.9590449633996014 /loss/FRt1/assignment_loss‚4.225600361824036 2loss/SRqqt3/assignment_loss‚1.2551921175898246 .loss/FRt3/detection_loss‚2.413544301308838 % _timestamp‚1.7884185500990026e+09 _step‚213 0loss/FRt2/assignment_loss‚3.8328472123001562³±RV†Ð 2loss/SRqqt4/assignment_loss‚1.0686847364382916 .loss/FBt1/detection_loss‚2.676278779655695 .loss/FBt3/detection_loss‚4.074291666348775 $loss/total_loss‚49.7148003513511 2loss/SRqqt1/assignment_loss‚1.8265606065190607 /loss/FRt1/detection_loss‚2.2544817298650743 epoch‚2 _runtime‚2049.836489739 0loss/FBt4/assignment_loss‚0.3037115617646267 .loss/FRt2/detection_loss‚2.706211948033535 0loss/SRqqt2/detection_loss‚2.224140871937076 1loss/SRqqt2/assignment_loss‚1.473756682088909 /loss/FRt4/detection_loss‚2.0043113227002323 0loss/SRqqt3/detection_loss‚1.786235196410485 .loss/FBt2/detection_loss‚3.232087880373001 trainer/global_step‚2599 /loss/FRt3/assignment_loss‚3.888651539297665 /loss/FRt4/assignment_loss‚2.036660080154737 0loss/FBt1/assignment_loss‚0.5572146886786526 0loss/FBt2/assignment_loss‚0.6194863064331002 .loss/FBt3/assignment_loss‚0.55522770992593 1loss/SRqqt1/detection_loss‚2.6869546953708894 1loss/SRqqt4/detection_loss‚1.0536233929329013 /loss/FBt4/detection_loss‚0.9590449633996014 /loss/FRt1/assignment_loss‚4.225600361824036 2loss/SRqqt3/assignment_loss‚1.2551921175898246 .loss/FRt3/detection_loss‚2.413544301308838 % _timestamp‚1.7884185500990026e+09 _step‚213 0loss/FRt2/assignment_loss‚3.8328472123001562Õáæ›Ï’‡‚(:‡ û³äÔ“Š²~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ÒTvˆ‚(:ë û³äÔ€’Ò€% disk./.usageGB‚255.00933456420898 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda.out‚ 208.3828125 disk.rbd0.in‚0( proc.memory.availableMB‚ 951.19921875 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚8837.91064453125* proc.memory.percent‚16.916370391845703 proc.cpu.threads‚17 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.113890409687425 disk.sda3.in‚ 126.77734375' disk.nvme0c0n1.out‚8837.91064453125 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.31640625 disk.nvme0c0n1.in‚ 4.31640625! proc.memory.rssMB‚ 2771.578125 network.recv‚824046 disk.drbd2080.out‚0" disk.dm-0.out‚8837.91064453125 disk.rbd1.out‚ 1.1005859375 disk.rbd0.out‚0 network.sent‚ 5.125901e+06 disk.dm-0.in‚ 4.31640625 disk.dm-1.out‚ 208.3828125 disk.sda2.in‚0 cpu‚25.345487398214534 disk.sda3.out‚ 208.3828125 disk.drbd2080.in‚0 disk.sda1.in‚0$ memory_percent‚94.19434070587158mÉ…‰‚(:’ Š´äÔØ‚~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ0}pŠ‚(:å Š´äÔÉÇ´€ disk.dm-1.out‚ 208.9453125 disk.rbd1.out‚ 1.1005859375 disk.nvme0n1.in‚ 4.31640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ memory_percent‚94.19288635253906 disk.rbd0.out‚0' disk.nvme0c0n1.out‚8880.53173828125 disk.sda2.out‚0$ disk./.usageGB‚255.0253028869629 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.31640625 disk.drbd1716.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 network.recv‚831602 disk.sda.in‚ 126.77734375 disk.sda.out‚ 208.9453125* disk./.usagePercent‚17.114962057225753 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0$ proc.memory.availableMB‚951.4375 cpu‚25.34349795624068 disk.dm-0.in‚ 4.31640625 disk.sda1.out‚0% disk.nvme0n1.out‚8880.53173828125 proc.cpu.threads‚17 network.sent‚ 5.234521e+06" disk.dm-0.out‚8880.53173828125 disk.dm-1.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 208.9453125àÂ÷‹‚(:’ ™´äÔçÓ~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à…þsŒ‚(:è ™´äÔÊ׈ disk.dm-0.in‚ 4.31640625 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 cpu‚25.341514782994736* disk./.usagePercent‚17.116681147818177 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.out‚8930.083984375% disk./.usageGB‚255.05091857910156 disk.rbd1.out‚ 1.16357421875$ memory_percent‚94.19445991516113 disk.nvme0n1.in‚ 4.31640625# disk.nvme0n1.out‚8930.072265625% disk.nvme0c0n1.out‚8930.060546875 disk.sda.out‚ 209.53515625! proc.memory.rssMB‚ 2771.578125 network.sent‚ 5.240558e+06 disk.sda2.out‚0 disk.dm-1.out‚ 209.53515625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.31640625 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 network.recv‚835763 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.916370391845703 disk.sda3.out‚ 209.53515625 disk.drbd2080.out‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 951.1796875¤t¹2- _wandbruntime‚2087  _runtime‚20878LrðÉŽÃ ' lr-AdamW/pg2‚0.00016150134197590385 trainer/global_step‚2649 % _timestamp‚1.7884185873168528e+09 _runtime‚2087.054358662 _step‚214 ' lr-AdamW/pg1‚0.00016150134197590385‹l­¶ÎÈ ' lr-AdamW/pg1‚0.00016150134197590385 ' lr-AdamW/pg2‚0.00016150134197590385 trainer/global_step‚2649 % _timestamp‚1.7884185873168528e+09 _runtime‚2087.054358662 _step‚214Ö[9‡_2- _wandbruntime‚2087  _runtime‚2087_ÔÛŽ ‘ÓO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170496349263360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35220614 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12750208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152168512965045 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚215 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49999098 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218036733952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209869897141 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168167452542464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156211742157824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ·“×Þ'’ØO TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50019135 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50019135 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88530689 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35220614 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38511552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12749772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12750208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152168512965045 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚215 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49999098 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168167428602368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218036733952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218036733952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2209869897141 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168167452542464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156211067792896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚2649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156211742157824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152169184442157 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149958660107264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156211742157824 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218693095936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156211067792896 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218693095936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2328920660992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170496349263360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088×/N2“- _wandbruntime‚2087  _runtime‚2087÷Âã&”ÝM trainer/global_step‚2649 _step‚216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884185880362618e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219512537088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156268240259584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211340770065 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156268914637824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218856161792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50037615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12754670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150013547198464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2329940615168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210686332105 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944‘HÀ¼è&•âM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2211340770065 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156268914637824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2218856161792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50037615 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12754670 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88563185 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38525568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22478574 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154049402100736 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150013547198464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88543120 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88563185 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2329940615168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12754237 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168229079340032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2210686332105 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚2649 _step‚216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38525568 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50017552 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219512537088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884185880362618e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2219512537088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38525570 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50017552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156268240259584 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170558850052608 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154049384097792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2329770712576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35232809 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚2699 $ _timestamp‚1.788418625648986e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2259678901760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2125.386800543 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156810701004288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12975635 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154954554832453 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚4142—Ý'¢×O aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171249965485568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50925150 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159069705541120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12975635 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2371793629696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156810701004288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152703866635264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156810683001344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35854863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159069705541120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173621944582656 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154954554832453 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22878806 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2259022539776 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159070379906048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159070379906048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2259022539776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚422 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152703849595904 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2259678901760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚2699 $ _timestamp‚1.788418625648986e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2259678901760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚2125.386800543 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156810701004288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39212354 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884186263589575e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2252176109473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152758753726464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚2699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 SDDeviceStatsMon̼·a itor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2126.097171705 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚220 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12980122 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39226368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22886974 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159127552386048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚220 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50943604 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35867540 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152758770765824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259841967616 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35867094 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12980122 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39226368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259841967616 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22886972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884186263589575e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90169972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156867054043136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90169972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171311616223232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173684257908736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155010946875297 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50943604 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39226370 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50963667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2372811588096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90190037 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39226368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50963667 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2252176109473 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90190037 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39226370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171311640163328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260498342912 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152758753726464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚156867036040192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156867054043136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2251521671513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚2699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159126878007808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155010275397977 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2126.097171705 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2372641685504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159628334943744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51851202 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159628352946688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159628352946688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚223 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2292195013709 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2291540575957 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157741268176973 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23287206 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51871239 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚2749 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161928343289344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚159628334943744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39913154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200147456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91784393 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51851202 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159628352946688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2414842698752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2414666491392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844ßõx2µ- _wandbruntime‚2163  _runtime‚2163˜ œã&¶ÝM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884186641001081e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2415524131840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51869656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155503960254464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159684687982592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2749 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2300827773440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2293011448881 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23295372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157796988742705 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _runtime‚2163.838351401 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155503960254464 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2300827773440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39927170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161986190134272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159684687982592 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚2749 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2300827773440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2301484148736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2293011448881 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23295372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174394177046528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2415689840128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39927170 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159684705985536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176809866886656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174394153106432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39927168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157796988742705 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161986190134272 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13207203 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51889719 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36502136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91816889 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91796824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161985515756032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91816889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157796317265385 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36502577 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 _runtime‚2163.838351401 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2415524131840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51869656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2301484148736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51889719 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13206764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176809677238272 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944àwêH2¸- _wandbruntime‚2163  _runtime‚2163ë¯ÐY¹Ó .loss/FRt1/assignment_loss‚3.57073040753603 0loss/FRt3/assignment_loss‚3.5836871993250963 2loss/SRqqt1/assignment_loss‚2.1184584516144933 0loss/FBt1/assignment_loss‚0.7826116214195793 1loss/FBt3/assignment_loss‚0.42421838928809547 1loss/SRqqt4/detection_loss‚1.1542911425729592 /loss/FBt4/detection_loss‚0.7718821649822141 0loss/FRt4/assignment_loss‚1.5792076169398792 1loss/SRqqt3/assignment_loss‚1.780833110075306 /loss/FRt3/detection_loss‚2.6269400025286327 /loss/FBt1/detection_loss‚2.3814921493331593 epoch‚2 _runtime‚2163.839544744 0loss/SRqqt2/assignment_loss‚1.94662666067164 1loss/FBt4/assignment_loss‚0.23578092721341967 /loss/FRt4/detection_loss‚1.6068054116085957 1loss/SRqqt1/detection_loss‚1.9305589255832492 1loss/SRqqt2/detection_loss‚2.2054000363705004 /loss/FRt2/assignment_loss‚3.159713113680482 /loss/FBt2/detection_loss‚2.6881386200586954 &loss/total_loss‚47.287074451020814 2loss/SRqqt4/assignment_loss‚1.0244865777591865 % _timestamp‚1.7884186641020794e+09 _step‚225 /loss/FRt2/detection_loss‚2.9408222483471036 1loss/SRqqt3/detection_loss‚1.8659549438291125 -loss/FBt3/detection_loss‚3.84559632341067 0loss/FBt2/assignment_loss‚0.6989928856610883 .loss/FRt1/detection_loss‚2.363845521211624 trainer/global_step‚2749ÆG%í^ºØ trainer/global_step‚2749 .loss/FRt1/assignment_loss‚3.57073040753603 0loss/FRt3/assignment_loss‚3.5836871993250963 2loss/SRqqt1/assignment_loss‚2.1184584516144933 0loss/FBt1/assignment_loss‚0.7826116214195793 1loss/FBt3/assignment_loss‚0.42421838928809547 1loss/SRqqt4/detection_loss‚1.1542911425729592 /loss/FBt4/detection_loss‚0.7718821649822141 0loss/FRt4/assignment_loss‚1.5792076169398792 1loss/SRqqt3/assignment_loss‚1.780833110075306 /loss/FRt3/detection_loss‚2.6269400025286327 /loss/FBt1/detection_loss‚2.3814921493331593 epoch‚2 _runtime‚2163.839544744 0loss/SRqqt2/assignment_loss‚1.94662666067164 1loss/FBt4/assignment_loss‚0.23578092721341967 /loss/FRt4/detection_loss‚1.6068054116085957 1loss/SRqqt1/detection_loss‚1.9305589255832492 1loss/SRqqt2/detection_loss‚2.2054000363705004 /loss/FRt2/assignment_loss‚3.159713113680482 /loss/FBt2/detection_loss‚2.6881386200586954 &loss/total_loss‚47.287074451020814 2loss/SRqqt4/assignment_loss‚1.0244865777591865 % _timestamp‚1.7884186641020794e+09 _step‚225 /loss/FRt2/detection_loss‚2.9408222483471036 1loss/SRqqt3/detection_loss‚1.8659549438291125 -loss/FBt3/detection_loss‚3.84559632341067 0loss/FBt2/assignment_loss‚0.6989928856610883 .loss/FRt1/detection_loss‚2.363845521211624á7Fóä’»‚(:‡ ó´äÔàæš gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.186' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,»;ð¼‚(:ç ó´äÔ£‹¿€ disk.drbd1716.out‚0! proc.memory.rssMB‚ 2771.578125 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.2265625% disk.nvme0n1.out‚9222.20068359375$ memory_percent‚97.08912372589111 network.sent‚ 5.480076e+06 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.in‚0 cpu‚25.329495319061483 network.recv‚865647 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 213.3203125( pr¨%M"‚oc.memory.availableMB‚ 476.91796875" disk.dm-0.out‚9222.20068359375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 213.3203125 disk.sda2.in‚0$ disk./.usageGB‚255.1399917602539 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.916370391845703 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚9222.20068359375* disk./.usagePercent‚17.122658923742687 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 213.3203125ZO½‚(:’ ‚µäÔÛ½¿~ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø Ò}t¾‚(:é ‚µäÔËìÔ€* proc.memory.percent‚16.916370391845703 disk.dm-1.in‚ 126.77734375' disk.nvme0c0n1.out‚9269.18505859375% disk./.usageGB‚255.16818237304688 disk.drbd2080.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 214.00390625 disk.sda1.out‚0 cpu‚25.327599452196345% disk.nvme0n1.out‚9269.18505859375 disk.sda3.out‚ 214.00390625 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.sda1.in‚0# memory_percent‚97.0900297164917 disk.drbd2266.in‚0 disk.sda2.out‚0 network.sent‚ 5.58888e+06 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625* disk./.usagePercent‚17.124550819420723 disk.drbd1716.in‚0" disk.dm-0.out‚9269.18505859375! proc.memory.rssMB‚ 2771.578125 network.recv‚872894 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 214.00390625( proc.memory.availableMB‚ 476.76953125u»£2¿- _wandbruntime‚2201  _runtime‚2201gÍÚÉÀà _step‚226 ' lr-AdamW/pg1‚0.00015995415000448306 ' lr-AdamW/pg2‚0.00015995415000448306 trainer/global_step‚2799 % _timestamp‚1.7884187014886336e+09 _runtime‚2201.226197137šÜÄÎÁÈ _runtime‚2201.226197137 _step‚226 ' lr-AdamW/pg1‚0.00015995415000448306 ' lr-AdamW/pg2‚0.00015995415000448306 trainer/global_step‚2799 % _timestamp‚1.7884187014886336e+09â`* 2Â- _wandbruntime‚2201  _runtime‚2201ÜS Ù'ÃÓO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160526638567269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2201.227167859 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52777254 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164787655402496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40613954 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341650513408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177415063192064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40613954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162446004889088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚2799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158194262651904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37125742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884187014893746e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40613952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568†”²Þ'ÄØO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 % _timestamp‚1.7884187014893746e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13429693 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2457586019328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40613952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164786981037568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177415039251968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915904 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52797291 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2332375915365 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164786981037568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160526638567269 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2201.227167859 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93391206 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52777254 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164787655402496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93411245 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162445986886144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164787655402496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158194279691264 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40613954 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162445986886144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2341650513408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177415063192064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40613954 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695606 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2457755935232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚227 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52797291 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2341650513408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162446004889088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚2799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162446004889088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160527310044381 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158194262651904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2340994151424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37125297 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179872625271296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37125742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30ãý¡@º2Å- _wandbruntime‚2201  _runtime‚2201R|‘<%ÆÝM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13434135 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179935322337280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93443741 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177476689989632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160582359132793 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164844153504256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2458444796416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177476713929728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2458608407552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158249166782464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164844827882496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52795708 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2201.918809145 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52815771 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884187021805778e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2342469954560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.curreTϛҧnt‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944By½è&ÇâM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52815771 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177476713929728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52795708 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37137907 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158249183821824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93423676 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2458608407552 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158249166782464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2341813579264 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13434586 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164844827882496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52795708 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2201.918809145 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52815771 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884187021805778e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2342469954560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162502357927936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2341813579264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2333846788289 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2799 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2342469954560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40627968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162502357927936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93423676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37138360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13434135 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179935322337280 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162502339924992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93443741 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93443741 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177476689989632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160582359132793 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2333192350329 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164844827882496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40627968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164844153504256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703774 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2458444796416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872äU Ä2È- _wandbruntime‚2201  _runtime‚2201Äv¶¯UÉÏ 2loss/SRqqt1/assignment_loss‚1.7098524804050859 0loss/FBt2/assignment_loss‚0.4255517357216579 1loss/FBt4/assignment_loss‚0.25476846694808514 .loss/FRt1/detection_loss‚2.467477770270528 /loss/FRt4/detection_loss‚1.7009866683433452 .loss/FBt3/detection_loss‚2.884485363960266 /loss/FRt2/assignment_loss‚4.382514123405729 1loss/SRqqt4/assignment_loss‚1.200386700178705 /loss/FRt3/detection_loss‚2.3290893727419326 $ _timestamp‚1.788418702182824e+09 0loss/FBt3/assignment_loss‚0.4378071935692181 1loss/SRqqt4/detection_loss‚1.2751702513934962 .loss/FBt1/detection_loss‚3.390185884454034 trainer/global_step‚2799 .loss/FRt3/assignment_loss‚3.27135715192678 /loss/FBt4/detection_loss‚0.9261254523819509 0loss/FRt1/assignment_loss‚3.9826161055951506 /loss/FRt4/assignment_loss‚2.123624296983083 2loss/SRqqt3/assignment_loss‚1.7377997881174088 0loss/FBt1/assignment_loss‚0.5334783933073016 1loss/SRqqt3/detection_loss‚1.9192430809140206 _step‚229 _runtime‚2201.920306351 2loss/SRqqt2/assignment_loss‚2.5112613240877786 1loss/SRqqt1/detection_loss‚2.2483906222356334 1loss/SRqqt2/detection_loss‚2.4635052410039036 %loss/total_loss‚50.56247524530738 epoch‚2 .loss/FRt2/detection_loss‚3.029777783368315 .loss/FBt2/detection_loss‚3.357019993993971öVR‘ZÊÔ .loss/FRt2/detection_loss‚3.029777783368315 .loss/FBt2/detection_loss‚3.357019993993971 2loss/SRqqt1/assignment_loss‚1.7098524804050859 0loss/FBt2/assignment_loss‚0.4255517357216579 1loss/FBt4/assignment_loss‚0.25476846694808514 .loss/FRt1/detection_loss‚2.467477770270528 /loss/FRt4/detection_loss‚1.7009866683433452 .loss/FBt3/detection_loss‚2.884485363960266 /loss/FRt2/assignment_loss‚4.382514123405729 1loss/SRqqt4/assignment_loss‚1.200386700178705 /loss/FRt3/detection_loss‚2.3290893727419326 $ _timestamp‚1.788418702182824e+09 0loss/FBt3/assignment_loss‚0.4378071935692181 1loss/SRqqt4/detection_loss‚1.2751702513934962 .loss/FBt1/detection_loss‚3.390185884454034 trainer/global_step‚2799 .loss/FRt3/assignment_loss‚3.27135715192678 /loss/FBt4/detection_loss‚0.9261254523819509 0loss/FRt1/assignment_loss‚3.9826161055951506 /loss/FRt4/assignment_loss‚2.123624296983083 2loss/SRqqt3/assignment_loss‚1.7377997881174088 0loss/FBt1/assignment_loss‚0.5334783933073016 1loss/SRqqt3/detection_loss‚1.9192430809140206 _step‚229 _runtime‚2201.920306351 2loss/SRqqt2/assignment_loss‚2.5112613240877786 1loss/SRqqt1/detection_loss‚2.2483906222356334 1loss/SRqqt2/detection_loss‚2.4635052410039036 %loss/total_loss‚50.56247524530738 epoch‚2åÄyÀ’Ë‚(:‡ ‘µäÔÜï‡~ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.827' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÀü¤uÌ‚(:ê ‘µäÔÁ¸ € disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚9322.55126953125* disk./.usagePercent‚17.126042063307672 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 network.recv‚877380 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.2265625 disk.sda1.out‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 476.73828125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125! proc.memory.rssMB‚ 2771.578125% disk./.usageGB‚255.19040298461914 disk.nvme0n1.in‚ 4.33203125% disk.nvme0n1.out‚9322.55126953125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9322.55126953125 disk.sda3.out‚ 214.33203125 disk.sda.out‚ 214.33203125 disk.sda1.in‚0$ memory_percent‚97.09022045135498 network.sent‚ 5.594803e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 214.33203125* proc.memory.percent‚16.916370391845703 cpu‚25.32563089938218 disk.rbd1.in‚0é7Ï ’Í‚(:‡  µäÔ”¡Œ~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'rM$u΂(:ê  µäÔÝô¢€ disk.sda.in‚ 126.77734375 disk.drbd2080.in‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.916370391845703 disk.rbd1.out‚ 1.2265625 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.sent‚ 5.703834e+06* disk./.usagePercent‚17.127665151074755$ disk./.usageGB‚255.2145881652832 disk.sda.out‚ 214.90234375 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 214.90234375 disk.drbd2265.out‚0" disk.dm-0.out‚9365.57080078125! proc.memory.rssMB‚ 2771.578125 network.recv‚884837 disk.dm-1.out‚ 214.90234375 disk.dm-0.in‚ 4.33203125% disk.nvme0n1.out‚9365.57080078125 disk.nvme0c0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚9365.57080078125$ memory_percent‚97.09169864654541( proc.memory.availableMB‚ 476.49609375 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 cpu‚25.323900962714752•% Ï‚(:’ ¯µäÔàû} gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚127.347' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.469399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s›P–hЂ(:Ý ¯µäÔ·ô€ disk.drbd2266.in‚0 disk.sda.out‚ 215.4453125) disk./.usagePercent‚17.12464119008031 disk.sda2.in‚0 disk.rbd1.out‚ 1.2265625! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 network.sent‚ 5.709108e+06 disk.drbd2080.in‚0 disk.drbd1716.out‚0% proc.memory.availableMB‚ 476.71875 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.sda3.out‚ 215.4453125 disk.dm-1.out‚ 215.4453125$ memory_percent‚97.09033966064453 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd1716.in‚0# disk.nvme0n1.out‚9419.115234375 network.recv‚888195 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚9419.115234375 disk.sda.in‚ 126.77734375% disk.nvme0c0n1.out‚9419.115234375% disk./.usageGB‚255.16952896118164 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 cpu‚25.32195881240881 disk.dm-1.in‚ 126.77734375O_£2Ñ- _wandbruntime‚2238  _runtime‚2238Ž9ƒÉÒà ' lr-AdamW/pg2‚0.00015940675835298415 trainer/global_step‚2849 % _timestamp‚1.7884187385733156e+09 _runtime‚2238.310834563 _step‚230 ' lr-AdamW/pg1‚0.00015940675835298415m(ÎÓÈ _step‚230 ' lr-AdamW/pg1‚0.00015940675835298415 ' lr-AdamW/pg2‚0.00015940675835298415 trainer/global_step‚2849 % _timestamp‚1.7884187385733156e+09 _runtime‚2238.310834563æãÖô|2Ô- _wandbruntime‚2238  _runtime‚2238+÷‰ËÙ'ÕÓO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2238.311751664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163312680434677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165263656831488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37762472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24104006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165263656831488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884187385739899e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2382636319232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2849 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167646293150720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30+»Z2ÖØO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160939469179904 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41314754 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53703306 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165263638828544 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24104004 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41314754 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182998333500416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚163313351911789 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2373865692525 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180497576135168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165263638828544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2500757365248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2381979957248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚2238.311751664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163312680434677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.sm^ÏÓ‚¬all_pool.current‚ 656361984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165263656831488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2381979957248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37762472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95038097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24104006 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2373211254773 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165263656831488 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2500910503936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53703306 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182998510579200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884187385739899e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53723343 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167646293150720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53723343 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2382636319232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 trainer/global_step‚2849 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167646293150720 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167645618785792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41314752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2239.01930103 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2382799385088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 trainer/global_step‚2849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2383455760384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 KRç&ÙáM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 2239.01930103 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37774684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163369072477521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167702791252480 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165319991867392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53721760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167703465630720 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160994390349824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163368401000201 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165320009870336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2382799385088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95050528 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165319991867392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 trainer/global_step‚2849 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95070593 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165320009870336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95070593 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41328768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180559250812928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160994373310464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2383455760384 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183061027635712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41328770 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53721760 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95050528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183060833793024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24112172 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13662512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2382799385088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2501606920192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24112174 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53741823 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872èÅ^AZ2Ú- _wandbruntime‚2239  _runtime‚2239¤úî¨TÛÎ /loss/FRt1/detection_loss‚2.3960255953398617 2loss/SRqqt1/assignment_loss‚2.0193191132059805 /loss/FBt4/detection_loss‚0.7803212936696466 _runtime‚2239.020408931 /loss/FRt2/assignment_loss‚4.030437921866392 .loss/FRt3/assignment_loss‚3.27914796575256 /loss/FRt2/detection_loss‚2.7390333673892875 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.710631848736243 2loss/SRqqt2/assignment_loss‚1.6930879680666981 /loss/FBt1/assignment_loss‚0.604440417987374 .loss/FBt1/detection_loss‚3.022470609708266 %loss/total_loss‚50.36579510991679 0loss/SRqqt4/detection_loss‚1.211102000812986 epoch‚2 _step‚233 0loss/FBt2/assignment_loss‚0.2920245704935951 0loss/FBt3/assignment_loss‚0.3780797695184851 0loss/FBt4/assignment_loss‚0.2768529515711006 .loss/FBt3/detection_loss‚3.070340527428521 1loss/SRqqt2/detection_loss‚2.1891011145056747 1loss/SRqqt3/detection_loss‚2.1580120328327883 $ _timestamp‚1.788418739282947e+09 /loss/FRt4/assignment_loss‚2.141130261696302 .loss/FRt3/detection_loss‚2.508926986352257 /loss/FBt2/detection_loss‚4.9071270525455475 2loss/SRqqt3/assignment_loss‚1.9351688736806745 2loss/SRqqt4/assignment_loss‚1.1532696884737086 /loss/FRt4/detection_loss‚1.9370782485661597 1loss/SRqqt1/detection_loss‚1.9326649297166754ȱçYÜÓ /loss/FRt2/assignment_loss‚4.030437921866392 .loss/FRt3/assignment_loss‚3.27914796575256 /loss/FRt2/detection_loss‚2.7390333673892875 trainer/global_step‚2849 /loss/FRt1/assignment_loss‚3.710631848736243 2loss/SRqqt2/assignment_loss‚1.6930879680666981 /loss/FBt1/assignment_loss‚0.604440417987374 .loss/FBt1/detection_loss‚3.022470609708266 %loss/total_loss‚50.36579510991679 0loss/SRqqt4/detection_loss‚1.211102000812986 epoch‚2 _step‚233 0loss/FBt2/assignment_loss‚0.2920245704935951 0loss/FBt3/assignment_loss‚0.3780797695184851 0loss/FBt4/assignment_loss‚0.2768529515711006 .loss/FBt3/detection_loss‚3.070340527428521 1loss/SRqqt2/detection_loss‚2.1891011145056747 1loss/SRqqt3/detection_loss‚2.1580120328327883 $ _timestamp‚1.788418739282947e+09 /loss/FRt4/assignment_loss‚2.141130261696302 .loss/FRt3/detection_loss‚2.508926986352257 /loss/FBt2/detection_loss‚4.9071270525455475 2loss/SRqqt3/assignment_loss‚1.9351688736806745 2loss/SRqqt4/assignment_loss‚1.1532696884737086 /loss/FRt4/detection_loss‚1.9370782485661597 1loss/SRqqt1/detection_loss‚1.9326649297166754 /loss/FRt1/detection_loss‚2.3960255953398617 2loss/SRqqt1/assignment_loss‚2.0193191132059805 /loss/FBt4/detection_loss‚0.7803212936696466 _runtime‚2239.020408931éCyJÝ‚(:’ ¾µäÔ”ì­~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.104' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.220800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lå#fÞ‚(:Û ¾µäÔŒ‘Ž network.recv‚892724 disk.sda.out‚ 216.09375% disk.nvme0c0n1.out‚9459.892578125 disk.drbd2265.in‚0 cpu‚25.320162028388186 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0# memory_percent‚97.0905065536499# disk.nvme0n1.out‚9459.892578125 disk.rbd0.out‚0 disk.drbd2265.out‚0% disk./.usageGB‚255.22197341918945 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 216.09375 network.sent‚ 5.715132e+06 disk.drbd2080.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 476.69140625 proc.cpu.threads‚17 disk.rbd1.out‚ 1.2265625 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125* disk./.usagePercent‚17.128160781661045 disk.drbd2080.out‚0 disk.dm-1.out‚ 216.09375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125* proc.memory.percent‚16.916370391845703 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚9459.892578125 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2771.578125²Æß‚(:’ ͵äÔÐì½~ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.762' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.352400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ISA‡oà‚(:ä ͵äÔ¾ÐÚ€ disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 216.69921875 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125 disk.sda.in‚ 126.77734375% disk.nvme0c0n1.out‚9507.642578125 disk.sda3.out‚ 216.69921875* proc.memory.percent‚16.916370391845703* disk./.usagePercent‚17.129344560499373 disk.sda.out‚ 216.69921875 disk.nvme0n1.in‚ 4.33203125 disk.drbd2265.out‚0 network.recv‚899283 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚97.09055423736572 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-0.out‚9507.642578125( proc.memory.availableMB‚ 476.68359375 disk.drbd1716.out‚0# disk.nvme0n1.out‚9507.642578125 cpu‚25.318402114536337$ disk./.usageGB‚255.2396125793457 disk.rbd1.out‚ 1.2265625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 network.sent‚ 5.823871e+06–›o2á- _wandbruntime‚2275  _runtime‚2275ÂMWÉâà % _timestamp‚1.7884187758744173e+09 _runtime‚2275.612056975 _step‚234 ' lr-AdamW/pg1‚0.00015884372358997877 ' lr-AdamW/pg2‚0.00015884372358997877 trainer/global_step‚2899XËÈÎãÈ ' lr-AdamW/pg1‚0.00015884372358997877 ' lr-AdamW/pg2‚0.00015884372358997877 trainer/global_step‚2899 % _timestamp‚1.7884187758744173e+09 _runtime‚2275.612056975 _step‚234êŠ0°2ä- _wandbruntime‚2275  _runtime‚2275àJdåÓO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96664949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512406 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54649395 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170504930898944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170504256534016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚168081290770944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170504256534016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166098722302085 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2543816887296 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13886583 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96664949 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42015552 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163684692747264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24512406 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168081308773888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423622125056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2422965763072 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42015554 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54649395 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38398989 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166155114344929 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170562103378944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414863029145 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54667875 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170561429000704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163739596877824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424441566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166154442867609 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183641787696128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ÿánþ éâM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38411332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13891205 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96697445 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ^JDeviceStatsMonitor.šÈ¶×on_train_batch_end/requested_bytes.large_pool.allocated‚163739596877824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424441566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029570 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166154442867609 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168137661812736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24520572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚183641787696128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168137661812736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96677380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚168137643809792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2544848871424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2423785190912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166155114344929 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163739579838464 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170562103378944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2414863029145 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96697445 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54667875 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186186636567552 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170561429000704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FËù’ñ‚(:‡ úµäÔ¬¿~ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b0¯×tò‚(:é úµäÔ¹þå€ disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.dm-1.out‚ 218.45703125 disk.drbd2266.out‚0# memory_percent‚97.0909833908081 disk.rbd0.out‚0" disk.dm-0.out‚9628.82177734375 disk.drbd1716.out‚0 network.sent‚ 5.944297e+06 disk.sda3.out‚ 218.45703125 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125( proc.memory.availableMB‚ 476.61328125 cpu‚25.313230018067838 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.916370391845703% disk./.usageGB‚255.27560424804688 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 218.45703125 disk.sda1.out‚0 disk.drbd1716.in‚0) disk./.usagePercent‚17.13175999158493' disk.nvme0c0n1.out‚9628.82177734375 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0n1.out‚9628.82177734375! proc.memory.rssMB‚ 2771.578125 network.recv‚914792 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2265625‹Hû2ó- _wandbruntime‚2312  _runtime‚2312èHômÇôÁ trainer/global_step‚2949 % _timestamp‚1.7884188129934144e+09 _runtime‚2312.730994516 _step‚238 & lr-AdamW/pg1‚0.0001582651630510677 & lr-AdamW/pg2‚0.0001582651630510677M|p2ÌõÆ _step‚238 & lr-AdamW/pg1‚0.0001582651630510677 & lr-AdamW/pg2‚0.0001582651630510677 trainer/global_step‚2949 % _timestamp‚1.7884188129934144e+09 _runtime‚2312.730994516îã$2ö- _wandbruntime‚2312  _runtime‚2312¼ÁSBÙ'÷ÓO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42716354 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42716352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39036066 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170898960716288 _runtime‚2312.732006446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2455536371341 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463951568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14114792 % _timestamp‚1.7884188129941742e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170898960716288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186662649901568 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55575447 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166429899275264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170898942713344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463951568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42716352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2586800247808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24920804 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42716354 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2464607930880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173362894282240 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42716352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39036066 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195953152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170898960716288 _runtime‚2312.732006446 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39035596 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2455536371341 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2464607930880 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2463951568896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14114792 % _timestamp‚1.7884188129941742e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170898960716288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186662649901568 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55575447 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14115260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189249646102528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173363568647168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166429899275264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55555410 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviŠÝºî ceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170898942713344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2463951568896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42716352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2586800247808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24920804 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 _runtime‚2313.438175798 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189311951428096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186724324579328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55593927 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173420066748928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 % _timestamp‚1.7884188137000628e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2455698368553 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2587822788608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2587650788864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39048348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173420066748928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170955313755136 % _timestamp‚1.7884188137000628e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55593927 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2455698368553 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170955295752192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚166484803405824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186724300639232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2587822788608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 _runtime‚2313.438175798 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55573864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173420741127168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2465427372032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98304232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14119846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚189311951428096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168940484735017 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2464770996736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98324297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24928972 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170955295752192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186724324579328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42730368 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2456352806513 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55593927 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166484786366464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4341ËOéÚ7154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173716594655744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39672286 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2350.165188253 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496371710749 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2495717272997 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14343509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884188504274085e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56501499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176221532030464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169175088763904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173716594655744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚243 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173716612658688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424{3¦MÞ'ˆØO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43417154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14343082 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173716594655744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39672286 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171670806036901 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25329206 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 _runtime‚2350.165188253 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176221532030464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2505593736704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99918653 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2496371710749 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2495717272997 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14343509 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884188504274085e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56501499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192375075411968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39672715 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚176221532030464 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629701062656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169175088763904 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173716594655744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚243 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173716612658688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192374887847424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176222206395392 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56501499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189745186784768 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43417152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2504937374720 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430óxB4ö2‰- _wandbruntime‚2350  _runtime‚2350.¡ºã&ŠÝM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14348180 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169230009933824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56519979 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884188511308153e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173772947694592 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173772947694592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56499916 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43431168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176279378875392 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99951149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14347747 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56499916 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2505756802560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43431170 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25337372 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171726526602425 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173772947694592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2506413177856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56499916 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165708288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43431168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169229992894464 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99931084 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99931084 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56519979 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 E=DeviceStatsMonitor.on®Ä#…_train_batch_end/inactive_split.all.peak‚908 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2496533707961 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚2999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43431168 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚244 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2506413177856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176279378875392 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56519979 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39685119 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884188511308153e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2497188145921 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173772965697536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844ôäÛŸ)2Œ- _wandbruntime‚2350  _runtime‚2350&ü{ÿ[Õ 2loss/SRqqt1/assignment_loss‚1.8484059712081624 2loss/SRqqt2/assignment_loss‚1.9753886610269547 /loss/FRt1/detection_loss‚2.5209511763909283 /loss/FBt4/detection_loss‚0.7339415703340186 trainer/global_step‚2999 0loss/FRt1/assignment_loss‚3.8879618153852573 %loss/total_loss‚50.56121132638878 2loss/SRqqt4/assignment_loss‚1.2651310448559958 0loss/FBt2/assignment_loss‚0.2614845221651185 0loss/FBt3/assignment_loss‚0.7296506042985129 /loss/FRt3/detection_loss‚2.7306602522730827 .loss/FRt4/detection_loss‚2.051888647467591 0loss/FRt2/assignment_loss‚3.6500456520489286 0loss/FRt3/assignment_loss‚3.3283247384760113 /loss/FBt1/assignment_loss‚1.233079481124878 .loss/FBt1/detection_loss‚4.388780854642391 2loss/SRqqt3/assignment_loss‚1.8731319568627938 1loss/SRqqt2/detection_loss‚2.1536627032540063 1loss/SRqqt4/detection_loss‚1.1775500143097866 /loss/FBt3/detection_loss‚2.5159344747662544 epoch‚2 $ _timestamp‚1.788418851132643e+09 _runtime‚2350.870137354 1loss/FBt4/assignment_loss‚0.14830566462966951 1loss/SRqqt1/detection_loss‚2.0211528534429117 .loss/FBt2/detection_loss‚3.535308673977852 /loss/FRt2/detection_loss‚2.8455080458096096 0loss/FRt4/assignment_loss‚1.6481253851291746 1loss/SRqqt3/detection_loss‚2.0368365625088867 _step‚245Úîkº`ŽÚ /loss/FBt4/detection_loss‚0.7339415703340186 trainer/global_step‚2999 0loss/FRt1/assignment_loss‚3.8879618153852573 %loss/total_loss‚50.56121132638878 2loss/SRqqt4/assignment_loss‚1.2651310448559958 0loss/FBt2/assignment_loss‚0.2614845221651185 0loss/FBt3/assignment_loss‚0.7296506042985129 /loss/FRt3/detection_loss‚2.7306602522730827 .loss/FRt4/detection_loss‚2.051888647467591 0loss/FRt2/assignment_loss‚3.6500456520489286 0loss/FRt3/assignment_loss‚3.3283247384760113 /loss/FBt1/assignment_loss‚1.233079481124878 .loss/FBt1/detection_loss‚4.388780854642391 2loss/SRqqt3/assignment_loss‚1.8731319568627938 1loss/SRqqt2/detection_loss‚2.1536627032540063 1loss/SRqqt4/detection_loss‚1.1775500143097866 /loss/FBt3/detection_loss‚2.5159344747662544 epoch‚2 $ _timestamp‚1.788418851132643e+09 _runtime‚2350.870137354 1loss/FBt4/assignment_loss‚0.14830566462966951 1loss/SRqqt1/detection_loss‚2.0211528534429117 .loss/FBt2/detection_loss‚3.535308673977852 /loss/FRt2/detection_loss‚2.8455080458096096 0loss/FRt4/assignment_loss‚1.6481253851291746 1loss/SRqqt3/detection_loss‚2.0368365625088867 _step‚245 2loss/SRqqt1/assignment_loss‚1.8484059712081624 2loss/SRqqt2/assignment_loss‚1.9753886610269547 /loss/FRt1/detection_loss‚2.5209511763909283õb)nœ‚(:‘ §¶äÔ÷¶Š~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.382000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËËJês‚(:è §¶äÔãöô€% disk./.usageGB‚255.29496002197266 disk.rbd0.in‚0 disk.sda2.out‚0" disk.dm-0.out‚9787.94189453125 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 220.0390625! proc.memory.rssMB‚ 2771.578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125% disk.nvme0n1.out‚9787.94189453125 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 220.0390625 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9787.94189453125 disk.dm-1.in‚ 126.77734375$ memory_percent‚97.09160327911377( proc.memory.availableMB‚ 476.51171875 disk.rbd1.out‚ 1.2265625* disk./.usagePercent‚17.133058973813664 disk.sda.in‚ 126.77734375 disk.sda.out‚ 220.0390625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 proc.cpu.threads‚17 disk.drbd2266.out‚0* proc.memory.percent‚16.916370391845703 cpu‚25.308148060343445 network.sent‚ 6.064414e+06 network.recv‚930025 disk.sda2.in‚0 disk.drbd2265.out‚0Jt‚¥‘‚(:… ¶¶äÔ”Üó~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êf—‚u’‚(:ê ¶¶äÔÊëÀ disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0% disk./.usageGB‚255.29962921142578 disk.rbd0.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2771.578125% disk.nvme0n1.out‚9831.06689453125 disk.sda2.out‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 220.58203125 disk.sda1.out‚0( proc.memory.availableMB‚ 476.24609375 proc.cpu.threads‚17 network.sent‚ 6.175591e+06 network.recv‚937811 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2265.out‚0$ memory_percent‚97.09322452545166* proc.memory.percent‚16.916370391845703* disk./.usagePercent‚17.133372327035577" disk.dm-0.out‚9831.06689453125 disk.rbd1.out‚ 1.2265625 disk.nvme0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚9831.06689453125 disk.sda3.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 220.58203125 disk.drbd1716.out‚0 disk.sda3.out‚ 220.58203125 disk.drbd2080.in‚0 cpu‚25.30654027912018kr7œ“‚(:‘ ŶäÔ¹ò˜~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.884' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.17679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ ¨t”‚(:é ŶäÔØ˘ disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.33203125% disk.nvme0n1.out‚9878.81494140625 network.sent‚ 6.179359e+06 disk.dm-1.out‚ 221.09765625 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2771.578125 cpu‚25.30492997878278 network.recv‚942123% disk./.usageGB‚255.30760192871094 disk.sda2.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 476.2578125* proc.memory.percent‚16.916370391845703 proc.cpu.threads‚17 disk.rbd1.out‚ 1.2265625 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 221.09765625 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.in‚0" disk.dm-0.out‚9878.81494140625 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9878.81494140625 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.sda1.out‚0 disk.drbd2080.out‚0* disk./.usagePercent‚17.133907382782137 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 221.09765625$ memory_percent‚97.09315299987793º†Àý2•- _wandbruntime‚2387  _runtime‚2387óÜÓ`È–Â ' lr-AdamW/pg1‚0.00015706195014054662 ' lr-AdamW/pg2‚0.00015706195014054662 trainer/global_step‚3049 % _timestamp‚1.7884188877585917e+09 _runtime‚ 2387.49612702 _step‚246RÐÕBÍ—Ç ' lr-AdamW/pg1‚0.00015706195014054662 ' lr-AdamW/pg2‚0.00015706195014054662 trainer/global_step‚3049 % _timestamp‚1.7884188877585917e+09 _runtime‚ 2387.49612702 _step‚246ö8rŒƒ2˜- _wandbruntime‚2387  _runtime‚2387uÅÝ'™×O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174456847904309 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2545923180544 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40309104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40309556 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57407514 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚2387.497014435 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57427551 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40309556 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2672936764416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57427551 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195500496807936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57407514 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚2387.497014435 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176534264601088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2546579542528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195500684372480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44117954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57427551 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14571500 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179080169778688 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176534246598144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44117954 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44131970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 trainer/global_step‚3049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25745774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57446031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176590599636992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14576467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2388.189951039 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884188884519625e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176590599636992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179137342245376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872@7¢ì&æM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176590617639936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2673791803904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171975216461824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2538023485329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101578001 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2547398983680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2546742608384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57425968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179137342245376 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40321790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2546742608384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101557936 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195563190149632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179138016623616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44131970 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195562996306944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2537369047369 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚174512568469833 trainer/global_step‚3049 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44131968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44131970 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57425968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25745774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14576018 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192889374405632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57446031 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44131968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192889398345728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902592 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101557936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101578001 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40322241 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176590599636992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2673621901312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14576467 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25745772 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚2388.189951039 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176590617639936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57446031 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7884188884519625e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176590599636992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171975199422464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928ø`ñžú2ž- _wandbruntime‚2388  _runtime‚2388[ëæYŸÓ .loss/FRt2/detection_loss‚2.803034826181829 1loss/SRqqt1/detection_loss‚1.9550288731401617 1loss/SRqqt3/detection_loss‚1.9081314736260817 trainer/global_step‚3049 % _timestamp‚1.7884188884534407e+09 _runtime‚2388.190876646 /loss/FRt2/assignment_loss‚4.011948101222515 0loss/FRt4/assignment_loss‚1.8473326049598993 1loss/FBt3/assignment_loss‚0.20527365486486815 %loss/total_loss‚47.64735606201863 2loss/SRqqt3/assignment_loss‚1.6606575363100722 0loss/FBt1/assignment_loss‚0.4469448097727483 0loss/FBt2/assignment_loss‚0.3746879158541553 .loss/FBt1/detection_loss‚2.788358980959112 _step‚249 2loss/SRqqt1/assignment_loss‚1.9275244719602844 0loss/SRqqt2/assignment_loss‚2.45526817237789 2loss/SRqqt4/assignment_loss‚1.1201801999257162 /loss/FRt1/detection_loss‚2.3480471083612153 0loss/FRt1/assignment_loss‚3.4686308430902884 1loss/SRqqt4/detection_loss‚1.3059376113689862 /loss/FBt4/detection_loss‚0.8908572917163082 /loss/FRt3/detection_loss‚2.3826054875497467 .loss/FBt3/detection_loss‚2.785525461037954 epoch‚2 0loss/FRt3/assignment_loss‚3.2371856183917433 0loss/FBt4/assignment_loss‚0.1975806422585443 /loss/FRt4/detection_loss‚1.9555817681784724 0loss/SRqqt2/detection_loss‚2.262548880143599 /loss/FBt2/detection_loss‚3.3084837287664413&=L^ Ø /loss/FRt1/detection_loss‚2.3480471083612153 0loss/FRt1/assignment_loss‚3.4686308430902884 1loss/SRqqt4/detection_loss‚1.3059376113689862 /loss/FBt4/detection_loss‚0.8908572917163082 /loss/FRt3/detection_loss‚2.3826054875497467 .loss/FBt3/detection_loss‚2.785525461037954 epoch‚2 0loss/FRt3/assignment_loss‚3.2371856183917433 0loss/FBt4/assignment_loss‚0.1975806422585443 /loss/FRt4/detection_loss‚1.9555817681784724 0loss/SRqqt2/detection_loss‚2.262548880143599 /loss/FBt2/detection_loss‚3.3084837287664413 .loss/FRt2/detection_loss‚2.803034826181829 1loss/SRqqt1/detection_loss‚1.9550288731401617 1loss/SRqqt3/detection_loss‚1.9081314736260817 trainer/global_step‚3049 % _timestamp‚1.7884188884534407e+09 _runtime‚2388.190876646 /loss/FRt2/assignment_loss‚4.011948101222515 0loss/FRt4/assignment_loss‚1.8473326049598993 1loss/FBt3/assignment_loss‚0.20527365486486815 %loss/total_loss‚47.64735606201863 2loss/SRqqt3/assignment_loss‚1.6606575363100722 0loss/FBt1/assignment_loss‚0.4469448097727483 0loss/FBt2/assignment_loss‚0.3746879158541553 .loss/FBt1/detection_loss‚2.788358980959112 _step‚249 2loss/SRqqt1/assignment_loss‚1.9275244719602844 0loss/SRqqt2/assignment_loss‚2.45526817237789 2loss/SRqqt4/assignment_loss‚1.1201801999257162ùlV0œ¡‚(:… Ô¶äÔÆýº~ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v!üs¢‚(:è Ô¶äÔ„äØ€ disk.sda.out‚ 221.5859375 disk.drbd2265.in‚0( proc.memory.availableMB‚ 476.47265625 proc.cpu.threads‚17* disk./.usagePercent‚17.131837049852734 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 network.sent‚ 6.185695e+06 network.recv‚946969 disk.dm-0.in‚ 4.33203125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 221.5859375 cpu‚25.303364455693337% disk./.usageGB‚255.27675247192383% disk.nvme0n1.out‚9921.32666015625 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0$ memory_percent‚97.09184169769287* proc.memory.percent‚16.916370391845703 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 221.5859375 disk.drbd1716.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9921.32666015625 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2771.578125" disk.dm-0.out‚9921.32666015625 disk.nvme0n1.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375ÿ°´r’£‚(:‡ ã¶äÔáô¨~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.9356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆP¸ï]¤‚(:Ò ã¶äÔΰ­ network.recv‚953260 disk.nvme0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚ 9974.640625 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 222.046875 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.2265625$ memory_percent‚97.09186553955078% proc.memory.availableMB‚ 476.46875 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.sent‚ 6.293684e+06 cpu‚25.301691134311103) disk./.usagePercent‚17.13448365574171 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.dm-1.out‚ 222.046875 disk.drbd1716.out‚0 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.916370391845703% disk./.usageGB‚255.31618881225586 disk.sda1.in‚0 disk.nvme0n1.out‚ 9974.640625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 222.046875! proc.memory.rssMB‚ 2771.578125 disk.nvme0c0n1.in‚ 4.33203125" disk.nvme0c0n1.out‚ 9974.640625 disk.sda2.in‚0 proc.cpu.threads‚17H0žâ2¥- _wandbruntime‚2424  _runtime‚2424É>S©É¦Ã ' lr-AdamW/pg1‚0.00015643754851670373 ' lr-AdamW/pg2‚0.00015643754851670373 trainer/global_step‚3099 % _timestamp‚1.7884189252317903e+09 _runtime‚2424.969314882 _step‚250ø¿ÑNÎ§È ' lr-AdamW/pg2‚0.00015643754851670373 trainer/global_step‚3099 % _timestamp‚1.7884189252317903e+09 _runtime‚2424.969314882 _step‚250 ' lr-AdamW/pg1‚0.00015643754851670373úÊÄÒœ2¨- _wandbruntime‚2424  _runtime‚2424)Ÿc©×O F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103152318 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198626230079488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2586908986368 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2424.970189292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179351916543488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2587565348352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3099 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26146004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195910260551168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58353603 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40944704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 F@DeviceStatsMonitoIÌ™7zr.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578042389565 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181939481891840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179351898540544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40945136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174665501819904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2586908986368 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44818752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181938807526912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577387951813 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2587565348352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44818752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58353603 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195910284491264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2424.970189292 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2715969528320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179351916543488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44818754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44818754 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2587565348352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚3099 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26146004 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195910260551168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2716128958464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58353603 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198626413449728 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40944704 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174665518859264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181938807526912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177242889771717 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26146006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578042389565 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58333566 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14799130 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181939481891840 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179351898540544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103172357 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ûn¸+Ä2«- _wandbruntime‚2425  _runtime‚2425訔|ç&¬áM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26154172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58352020 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 trainer/global_step‚3099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021²@ h!­æM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103204853 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103204853 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181996654371840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26154172 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44832768 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58352020 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174720422989824 trainer/global_step‚3099 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2588384789504 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40957449 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103184788 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195971935228928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179408251579392 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174720405950464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40957906 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179408269582336 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2588384789504 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198688722385920üò!€2®- _wandbruntime‚2425  _runtime‚2425ZùDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201751891295232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184797445275136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41581240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184798119640064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2618223291221 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59279655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59259618 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198992797434368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928(pzêâ'¼ÜO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2628551154176 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚184797445275136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2759093860864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177410725387264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201752076762624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182169568485888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41581240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26554406 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚184798119640064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26554404 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2618223291221 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45519552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59279655 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59259618 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198992797434368 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚184798119640064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104799209 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45519554 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182169550482944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104779170 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59279655 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182169550482944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180029603116237 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚2462.524418161 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2628551154176 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚3149 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2627894792192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2627894792192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2759255388160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45519554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180028931639125 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161527296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45519552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚184797445275136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182169568485888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201751891295232ÿ¤ssÒ2½- _wandbruntime‚2463  _runtime‚2463ºêÛfA ¾àM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2629370595328 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182225903521792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26562572 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚3149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177465612478464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45533570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocatÙj¥ed_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2629370595328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15031753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619039726185 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104811640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2760117592576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 2463.22011069 _step‚256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59278072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884189634819129e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45533568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201814402153472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59278072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41593875 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884189634819129e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180085323681969 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59298135 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚184855292120064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26562574 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚3149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚177465612478464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2619694164145 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182225903521792 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45533570 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41594327 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201814589704704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2628714220032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199054472112128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2629370595328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15031753 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15031303 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104811640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182225921524736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199054448172032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2619039726185 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104831705 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104811640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2760117592576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59278072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 2463.22011069 _step‚256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42217632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26962804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60185670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202075358257664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180155914875904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2499.598188137 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2669536960000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_sÊáÁäÈtart/inactive_split_bytes.all.allocated‚204877728385536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42217170 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952æ6áá'ÌÛO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46220352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2668880598016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182814973506533 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180155931915264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187656083023360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2659713068381 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187656757388288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46220354 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60185670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202075358257664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46220354 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187656083023360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187656757388288 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60185670 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180155914875904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚2499.598188137 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204877526140928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304512 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2669536960000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204877728385536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2802370127872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42217170 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202075334317568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184987220428288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26962806 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106406022 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2668880598016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42217632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182815644983645 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2802191823360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46220352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26962804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15254826 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15254366 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106438492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60204124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15259304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180210836045824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803214157312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803048449024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182870694072057 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187713929868288 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185043573467136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185043555464192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185043573467136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185043555464192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106438492 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189648384 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42229839 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60204124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26970974 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15259304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204940033504256 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670356401152 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180210836045824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106458557 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2803214157312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2803048449024 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60204124 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202137008995328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182870694072057 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚3199„mòpÉ2Ð- _wandbruntime‚2500  _runtime‚2500ïC¾]Ñ× 0loss/FRt3/assignment_loss‚3.7853106224536894 1loss/FBt2/assignment_loss‚0.12753009479355468 trainer/global_step‚3199 1loss/FBt4/assignment_loss‚0.24936077216142769 /loss/FRt3/detection_loss‚2.3405885088443754 1loss/SRqqt3/detection_loss‚2.0583854764699936 0loss/FRt4/assignment_loss‚1.7536744149401784 0loss/FBt3/assignment_loss‚0.5462146200625284 .loss/FRt4/detection_loss‚1.703994554351084 1loss/SRqqt4/detection_loss‚1.1318318723933771 .loss/FBt1/detection_loss‚2.898582196235657 /loss/FBt4/detection_loss‚0.7630883447433773 _runtime‚2500.302740391 /loss/FRt2/detection_loss‚2.7458701711148024 2loss/SRqqt1/assignment_loss‚1.7027436584307283 2loss/SRqqt2/assignment_loss‚1.5771725640321772 2loss/SRqqt4/assignment_loss‚0.9711465329455677 /loss/FRt1/detection_loss‚2.2478390102801113 /loss/FBt3/detection_loss‚2.8517181426286697 epoch‚2 $ _timestamp‚1.788419000565312e+09 _step‚261 0loss/FRt2/assignment_loss‚3.7240068838000298 2loss/SRqqt3/assignment_loss‚1.7168811131620183 0loss/FRt1/assignment_loss‚3.9092167641805564 /loss/FBt2/detection_loss‚3.6205825905005136 $loss/total_loss‚46.6718249533597 1loss/FBt1/assignment_loss‚0.33190148957003773 1loss/SRqqt1/detection_loss‚1.7489817811577928 0loss/SRqqt2/detection_loss‚2.165202774107456>ü™f‹ÒÜ 1loss/SRqqt1/detection_loss‚1.7489817811577928 0loss/SRqqt2/detection_loss‚2.165202774107456 0loss/FRt3/assignment_loss‚“u“Z×3.7853106224536894 1loss/FBt2/assignment_loss‚0.12753009479355468 trainer/global_step‚3199 1loss/FBt4/assignment_loss‚0.24936077216142769 /loss/FRt3/detection_loss‚2.3405885088443754 1loss/SRqqt3/detection_loss‚2.0583854764699936 0loss/FRt4/assignment_loss‚1.7536744149401784 0loss/FBt3/assignment_loss‚0.5462146200625284 .loss/FRt4/detection_loss‚1.703994554351084 1loss/SRqqt4/detection_loss‚1.1318318723933771 .loss/FBt1/detection_loss‚2.898582196235657 /loss/FBt4/detection_loss‚0.7630883447433773 _runtime‚2500.302740391 /loss/FRt2/detection_loss‚2.7458701711148024 2loss/SRqqt1/assignment_loss‚1.7027436584307283 2loss/SRqqt2/assignment_loss‚1.5771725640321772 2loss/SRqqt4/assignment_loss‚0.9711465329455677 /loss/FRt1/detection_loss‚2.2478390102801113 /loss/FBt3/detection_loss‚2.8517181426286697 epoch‚2 $ _timestamp‚1.788419000565312e+09 _step‚261 0loss/FRt2/assignment_loss‚3.7240068838000298 2loss/SRqqt3/assignment_loss‚1.7168811131620183 0loss/FRt1/assignment_loss‚3.9092167641805564 /loss/FBt2/detection_loss‚3.6205825905005136 $loss/total_loss‚46.6718249533597 1loss/FBt1/assignment_loss‚0.33190148957003773…í9¥’‘Ó‚(:† ½·äÔÎù”~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚135.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.031 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}w¹ÎqÔ‚(:æ ½·äÔŸŠ«€ disk.sda3.out‚ 225.296875 disk.drbd2080.in‚0 disk.drbd2080.out‚0# memory_percent‚97.0958948135376 disk.dm-0.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2771.578125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚10285.38720703125( proc.memory.availableMB‚ 475.80859375# disk.dm-0.out‚10285.38720703125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.916370391845703 network.sent‚ 6.533725e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 225.296875 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 cpu‚25.292663957979112 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 network.recv‚984197* disk./.usagePercent‚17.139488859043645$ disk./.usageGB‚255.3907699584961 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 225.296875 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd1.out‚ 1.2265625 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚10285.38720703125 disk.rbd0.out‚0Nß`Ú‘Õ‚(:† Ì·äÔÅëý} gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ruÀsÖ‚(:è Ì·äÔî“€ disk.rbd1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 475.80859375* proc.memory.percent‚16.916370391845703# disk.dm-0.out‚10312.03173828125 disk.sda.in‚ 126.77734375 network.sent‚ 6.643001e+06* disk./.usagePercent‚17.141169036490865 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 225.7734375 disk.sda.out‚ 225.7734375# memory_percent‚97.0958948135376( disk.nvme0c0n1.out‚10312.03173828125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2771.578125 cpu‚25.29133654934601 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 1.2265625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 225.7734375 network.recv‚991920$ disk./.usageGB‚255.4158058166504 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚10312.03173828125 disk.drbd2265.in‚0 disk.drbd2080.in‚0¹?æׂ͒(:‡ Û·äÔÃň~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&€|ZzØ‚(:ï Û·äԮШ€ disk.drbd2266.out‚0 disk.sda.out‚ 226.33203125 disk.sda1.in‚0 disk.sda3.out‚ 226.33203125 disk.nvme0c0n1.in‚ 4.33203125 network.sent‚ 6.651494e+06 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 226.33203125 disk.rbd1.out‚ 1.2265625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚ 1.001833e+06* disk./.usagePercent‚17.141832096004546 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚10360.7451171875 disk.sda3.in‚ 126.77734375$ memory_percent‚97.09591865539551 cpu‚25.28991952461164 disk.drbd2080.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚10360.7451171875 disk.nvme0n1.in‚ 4.33203125' proc.memory.availableMB‚ 475.8046875* proc.memory.percent‚16.916370391845703 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0% disk.nvme0n1.out‚10360.7451171875 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375! proc.memory.rssMB‚ 2771.578125% disk./.usageGB‚255.42568588256836pRÜÈ2Ù- _wandbruntime‚2537  _runtime‚2537à = ÉÚà _runtime‚2537.114016413 _step‚262 ' lr-AdamW/pg1‚0.00015447473377433555 ' lr-AdamW/pg2‚0.00015447473377433555 trainer/global_step‚3249 % _timestamp‚1.7884190373763926e+09Ad ¿ÎÛÈ ' lr-AdamW/pg1‚0.00015447473377433555 ' lr-AdamW/pg2‚0.00015447473377433555 trainer/global_step‚3249 % _timestamp‚1.7884190373763926e+09 _runtime‚2537.114016413 _step‚262†ìËw2Ü- _wandbruntime‚2537  _runtime‚2537«ðTÝ'Ý×O JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185601015373941 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187804854367744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2699893970037 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚2537.114873211 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844…¯%çâ'ÞÜO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15481654 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2710522765824 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181273088 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚3249 % _timestamp‚1.7884190373770773e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190515395136512 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61111722 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205157871200768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61111722 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚187804854367744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27371206 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185601015373941 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187804854367744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2845495917056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2699893970037 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2709866403840 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61131759 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190514720771584 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚187804872370688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚190515395136512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42852411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15481207 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108052913 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚2537.114873211 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2845338584064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108032874 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42852860 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2846367744000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2700710405001 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚2537.811765284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153125376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2846214618624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61150239 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788419038073587e+09 _step‚264 NADeviceStatsMonitor.on_train_batch_end/allocatioÿת$n.small_pool.freed‚61130176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208065913622528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190572567616512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208065736557056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198Qêyë&áåM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46935170 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2710685831680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46935170 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚208065913622528 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205219521938432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187861207406592 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185656735939465 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190572567616512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187861225409536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208065736557056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205219545878528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15486198 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46935168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61150239 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42865572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2846367744000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108065344 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2710685831680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108065344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42865119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2700710405001 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187861207406592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚2537.811765284 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153125376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61130176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27379374 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187861225409536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15485747 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108085409 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2846214618624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185657407416785 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61150239 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788419038073587e+09 _step‚264 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61130176 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190571893238272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211128818691072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47621954 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2750852209664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47621952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193374032884736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27779604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15708456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185646344971264o‰¹îÜO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62057811 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15708885 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2751508571648 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚208240408083968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27779604 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211128639515136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188387057241349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190622506310144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188387728718461 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15708456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185646344971264 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2888231431168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190622524313088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62057811 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235840 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193373358519808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211128818691072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47621954 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2750852209664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47621952 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62037774 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2888386667008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190622524313088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190622506310144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109659726 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47621954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109679765 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47621952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193374032884736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2750852209664 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2574.959797547 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193431205364736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190678877351936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211191337303040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788419075221664e+096DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208302058821632 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190678877351936 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43500715 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109692196 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43501160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185701232062464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190678859348992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2742200182369 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47635968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 $ _timestamp‚1.788419075221664e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193431205364736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47635970 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47635970 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188443449284193 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185701249101824 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2751671637504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27787772 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188442777806873 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2752328012800 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109712261 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2889254541312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193430530986496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211191151848960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2574.959797547Œn+™‰2ò- _wandbruntime‚2574  _runtime‚2574R±ÅÎ\óÖ /loss/FBt2/detection_loss‚3.2740847137239246 epoch‚2 2loss/SRqqt1/assignment_loss‚1.7568325031288834 2loss/SRqqt3/assignment_loss‚1.7974706145739832 /loss/FRt3/detection_loss‚2.1562092091356004 1loss/SRqqt2/detection_loss‚2.2669428948845183 _step‚269 /loss/FRt3/assignment_loss‚3.742807704423155 /loss/FRt4/detection_loss‚1.6980497358100755 /loss/FBt4/detection_loss‚0.9272816596659705 0loss/FRt1/assignment_loss‚3.1185817380746204 1loss/SRqqt4/detection_loss‚1.3250672040601907 trainer/global_step‚3299 1loss/SRqqt2/assignment_loss‚2.250484651398091 1loss/SRqqt4/assignment_loss‚1.244945323031585 1loss/FBt3/assignment_loss‚0.07835299987345934 1loss/SRqqt1/detection_loss‚1.9651618215233781 %loss/total_loss‚48.14646686720177 /loss/FBt1/assignment_loss‚0.524008414811555 1loss/FBt2/assignment_loss‚0.39790930638991995 .loss/FRt2/detection_loss‚2.991765974029418 _runtime‚2574.960860633 0loss/FRt4/assignment_loss‚1.8825200253299303 1loss/SRqqt3/detection_loss‚1.9115149257488029 /loss/FBt3/detection_loss‚2.7886402351515636 % _timestamp‚1.7884190752233999e+09 1loss/FBt4/assignment_loss‚0.13397298322776935 /loss/FRt1/detection_loss‚2.6941712220509846 /loss/FBt1/detection_loss‚3.0529348386658564 /loss/FRt2/assignment_loss‚4.166756168488534÷÷ÀaôÛ 1loss/SRqqt4/detection_loss‚1.3250672040601907 trainer/global_step‚3299 1loss/SRqqt2/assignment_loss‚2.250484651398091 1loss/SRqqt4/assignment_loss‚1.244945323031585 1loss/FBt3/assignment_loss‚0.07835299987345934 1loss/SRqqt1/detection_loss‚1.9651618215233781 %loss/total_loss‚48.14646686720177 /loss/FBt1/assignment_loss‚0.524008414811555 1loss/FBt2/assignment_loss‚0.39790930638991995 .loss/FRt2/detection_loss‚2.991765974029418 _runtime‚2574.960860633 0loss/FRt4/assignment_loss‚1.8825200253299303 1loss/SRqqt3/detection_loss‚1.9115149257488029 /loss/FBt3/detection_loss‚2.7886402351515636 % _timestamp‚1.7884190752233999e+09 1loss/FBt4/assignment_loss‚0.13397298322776935 /loss/FRt1/detection_loss‚2.6941712220509846 /loss/FBt1/detection_loss‚3.0529348386658564 /loss/FRt2/assignment_loss‚4.166756168488534 /loss/FBt2/detection_loss‚3.2740847137239246 epoch‚2 2loss/SRqqt1/assignment_loss‚1.7568325031288834 2loss/SRqqt3/assignment_loss‚1.7974706145739832 /loss/FRt3/detection_loss‚2.1562092091356004 1loss/SRqqt2/detection_loss‚2.2669428948845183 _step‚269 /loss/FRt3/assignment_loss‚3.742807704423155 /loss/FRt4/detection_loss‚1.6980497358100755 /loss/FBt4/detection_loss‚0.9272816596659705 0loss/FRt1/assignment_loss‚3.1185817380746204v¤k±’õ‚(:‡ ˆ¸äÔœ½È gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07¨óyö‚(:î ˆ¸äÔÌÀ¶€ disk.sda3.in‚ 126.77734375$ memory_percent‚97.09641933441162( proc.memory.availableMB‚ 475.72265625$ disk./.usageGB‚255.4496841430664 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.dm-0.in‚ 4.33203125 disk.rbd0.in‚0 cpu‚25.28587056549712 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.916370391845703 proc.cpu.threads‚17 network.sent‚ 6.772461e+06# disk.dm-0.out‚10494.48681640625 disk.rbd0.out‚0 disk.dm-1.out‚ 227.8984375 disk.sda3.out‚ 227.8984375) disk./.usagePercent‚17.14344263940245 disk.drbd2080.out‚0 disk.rbd1.out‚ 1.2265625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 227.8984375 disk.sda1.in‚0 network.recv‚ 1.017503e+06& disk.nvme0n1.out‚10494.48681640625( disk.nvme0c0n1.out‚10494.48681640625 disk.sda2.in‚0! proc.memory.rssMB‚ 2771.578125 disk.nvme0c0n1.in‚ 4.33203125ÜWc”÷‚(:‡ —¸äÔÞì†~ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741~~Y»þ678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.734' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.1468 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º~ñuø‚(:ê —¸äÔ‡§ž€ network.sent‚ 6.880433e+06% disk./.usageGB‚255.45570373535156 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 1.2265625 disk.rbd0.out‚0 disk.dm-0.in‚ 4.33203125# disk.dm-0.out‚10557.04150390625 disk.nvme0n1.in‚ 4.33203125 cpu‚25.28452460353172* disk./.usagePercent‚17.143846619291484( disk.nvme0c0n1.out‚10557.04150390625 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0! proc.memory.rssMB‚ 2771.578125 disk.sda1.out‚0 network.recv‚ 1.023128e+06 disk.sda3.out‚ 228.453125 disk.drbd2266.in‚0% proc.memory.availableMB‚ 475.71875* proc.memory.percent‚16.916370391845703 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 228.453125 disk.drbd2266.out‚0 disk.sda.out‚ 228.453125 disk.drbd1716.in‚0 disk.sda2.in‚0& disk.nvme0n1.out‚10557.04150390625$ memory_percent‚97.09644317626953 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 disk.rbd1.in‚0 disk.drbd1716.out‚0WñçS’ù‚(:‡ ¦¸äÔ¡Ÿª~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.864' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'›°xú‚(:í ¦¸äÔÏÀ€ disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0$ memory_percent‚97.09680080413818 cpu‚25.283184691739525* disk./.usagePercent‚17.144568816545576 disk.sda3.out‚ 229.2421875 disk.rbd1.in‚0( proc.memory.availableMB‚ 475.66015625* proc.memory.percent‚16.916370391845703 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0# disk.dm-0.out‚10595.95166015625 disk.dm-1.out‚ 229.2421875 disk.rbd1.out‚1.3125 disk.drbd1716.in‚0 network.recv‚ 1.027752e+06( disk.nvme0c0n1.out‚10595.95166015625 disk.sda.out‚ 229.2421875 network.sent‚ 6.886395e+06 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚10595.95166015625 disk.nvme0c0n1.in‚ 4.33203125$ disk./.usageGB‚255.4664649963379ªüŸg2û- _wandbruntime‚2611  _runtime‚2611i!LqÈü $ _timestamp‚1.788419111953642e+09 _runtime‚2611.691201417 _step‚270 ' lr-AdamW/pg1‚0.00015309288689433455 ' lr-AdamW/pg2‚0.00015309288689433455 trainer/global_step‚3349„¨jkÍýÇ ' lr-AdamW/pg1‚0.00015309288689433455 ' lr-AdamW/pg2‚0.00015309288689433455 trainer/global_step‚3349 $ _timestamp‚1.788419111953642e+09 _runtime‚2611.691201417 _step‚270Ž6e4¸2þ- _wandbruntime‚2611  _runtime‚2611 åRÜ'ÿÖO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48322752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191173770585869 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191173099108757 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196231996268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214254254985728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62983863 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62963826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211322944967168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193440158252544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211322968907264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48322754 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚3349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788419111954337e+09 _runtime‚2611.692099833 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2C€`á'€ÛO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44125494 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111306617 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48322752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2792494377472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚193440176255488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191173770585869 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62983863 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214254438355968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191173099108757 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196231996268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159430144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183370240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15937046 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44125050 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214254254985728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62983863 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2781564648853 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48322752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚193440176255488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2791838015488 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62963826 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28188006 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2931469448704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2931310018560 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211322944967168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188391534459904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2791838015488 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196232670632960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188391551499264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111286578 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚193440158252544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211322968907264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2792494377472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48322754 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196231996268032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚3349 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48322754 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788419111954337e+09 _runtime‚2611.692099833 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28188004 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62963826 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚196232670632960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2782219086605 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576Ž uu2- _wandbruntime‚2612  _runtime‚2612©w–"G"‚àM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 _step‚272 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196289843112960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2792657443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2932178795008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62982280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2793313818624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2612.415219471 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48336768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3349 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191229491151601 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2782381083817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 $ _timestamp‚1.788419112677228e+09 H:De&«€áŸviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0šuzë&ƒåM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2932336114688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚196289843112960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2792657443328 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191228819674281 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚193496511291392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211384595704832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2932178795008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62982280 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2793313818624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44137788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188446438590464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2612.415219471 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48336768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3349 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚193496511291392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15941616 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188446455629824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191229491151601 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111339113 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44138233 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2782381083817 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚193496529294336 $ _timestamp‚1.788419112677228e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111339113 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48336768 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63002343 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214316955759616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111319048 _step‚272¹Þª2„- _wandbruntime‚2612  _runtime‚2612«„÷ Y…Ó /loss/FRt3/assignment_loss‚3.188118427227705 1loss/SRqqt1/detection_loss‚2.0336479943159014 /loss/FBt2/detection_loss‚3.4197377496295505 .loss/FBt3/detection_loss‚2.549254055206592 0loss/FRt4/assignment_loss‚1.9655044531380688 /loss/FRt1/assignment_loss‚3.010077145169763 /loss/FRt2/assignment_loss‚2.399246744811535 1loss/SRqqt1/assignment_loss‚1.535519573750884 2loss/SRqqt3/assignment_loss‚1.6826183455330985 1loss/SRqqt3/detection_loss‚1.8115037056256313 epoch‚2 trainer/global_step‚3349 _step‚273 /loss/FRt1/detection_loss‚2.2095926439060882 /loss/FRt2/detection_loss‚3.7908394783735275 /loss/FBt4/detection_loss‚0.6668685406058406 0loss/FBt3/assignment_loss‚0.5527587326673361 1loss/SRqqt2/detection_loss‚2.4127880568439894 /loss/FBt1/detection_loss‚2.7266322188079357 &loss/total_loss‚44.890509540917634 $ _timestamp‚1.788419112678929e+09 1loss/SRqqt2/assignment_loss‚1.302145915256964 0loss/FBt2/assignment_loss‚0.3815587001218874 1loss/FBt4/assignment_loss‚0.22815685323953158 .loss/FRt3/detection_loss‚2.655588612342492 .loss/FRt4/detection_loss‚1.722423392382485 1loss/SRqqt4/detection_loss‚0.9533263545640758 2loss/SRqqt4/assignment_loss‚1.2205317569240717 1loss/FBt1/assignment_loss‚0.47207009047267423 _runtime‚2612.416394308}¬g^†Ø /loss/FRt3/assignment_loss‚3.188118427227705 1loss/SRqqt1/detection_loss‚2.0336479943159014 /loss/FBt2/detection_loss‚3.4197377496295505 .loss/FBt3/detection_loss‚2.549254055206592 0loss/FRt4/assignment_loss‚1.9655044531380688 /loss/FRt1/assignment_loss‚3.010077145169763 /loss/FRt2/assignment_loss‚2.399246744811535 1loss/SRqqt1/assignment_loss‚1.535519573750884 2loss/SRqqt3/assignment_loss‚1.6826183455330985 1loss/SRqqt3/detection_loss‚1.8115037056256313 epoch‚2 trainer/global_step‚3349 _step‚273 /loss/FRt1/detection_loss‚2.2095926439060882 /loss/FRt2/detection_loss‚3.7908394783735275 /loss/FBt4/detection_loss‚0.6668685406058406 0loss/FBt3/assignment_loss‚0.5527587326673361 1loss/SRqqt2/detection_loss‚2.4127880568439894 /loss/FBt1/detection_loss‚2.7266322188079357 &loss/total_loss‚44.890509540917634 $ _timestamp‚1.788419112678929e+09 1loss/SRqqt2/assignment_loss‚1.302145915256964 0loss/FBt2/assignment_loss‚0.3815587001218874 1loss/FBt4/assignment_loss‚0.22815685323953158 .loss/FRt3/detection_loss‚2.655588612342492 .loss/FRt4/detection_loss‚1.722423392382485 1loss/SRqqt4/detection_loss‚0.9533263545640758 2loss/SRqqt4/assignment_loss‚1.2205317569240717 1loss/FBt1/assignment_loss‚0.47207009047267423 _runtime‚2612.416394308‘kÚ伇‚(:… µ¸äÔÞ±~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.388 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gy­Árˆ‚(:ç µ¸äÔž«Ç€ disk.dm-1.out‚ 230.0390625 disk.drbd2266.in‚0 disk.sda.out‚ 230.0390625 disk.nvme0c0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚10640.10986328125 disk.drbd2080.out‚0% disk./.usageGB‚255.47288131713867 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0& disk.nvme0n1.out‚10640.10986328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0# disk.dm-0.out‚10640.10986328125 disk.drbd2266.out‚0 disk.rbd1.out‚1.375# proc.memory.availableMB‚475.625 proc.cpu.threads‚17 network.sent‚ 6.89233e+06 network.recv‚ 1.031733e+06* disk./.usagePercent‚17.144999421218166 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 230.0390625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚97.09701538085938! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 cpu‚25.281864271235374 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0@£Ê‰‚(:… ĸäÔË÷º~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ»^ÆuŠ‚(:ê ĸäÔòþဠdisk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.038626e+06$ disk./.usageGB‚255.4775047302246 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚10681.80517578125 disk.sda3.out‚ 230.50390625 disk.drbd2265.in‚0$ memory_percent‚97.09854125976562! proc.memory.rssMB‚ 2771.578125 disk.drbd2266.in‚0& disk.nvme0n1.out‚10681.80517578125 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚1.375# proc.memory.availableMB‚475.375 network.sent‚ 7.001308e+06* disk./.usagePercent‚17.145309702349664 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda.out‚ 230.50390625 cpu‚25.280555778783427 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 230.50390625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.dm-0.in‚ 4.33203125# disk.dm-0.out‚10681.80517578125* proc.memory.percent‚16.916370391845703 disk.sda2.in‚0g$§D2‹- _wandbruntime‚2649  _runtime‚2649g¦ÌÇŒÁ & lr-AdamW/pg1‚0.0001523803997413648 & lr-AdamW/pg2‚0.0001523803997413648 trainer/global_step‚3399 % _timestamp‚1.7884191492924638e+09 _runtime‚2649.030011361 _step‚274&æ5ÌÆ trainer/global_step‚3399 % _timestamp‚1.7884191492924638e+09 _runtime‚2649.030011361 _step‚274 & lr-AdamW/pg1‚0.0001523803997413648 & lr-AdamW/pg2‚0.0001523803997413648’û½ ›2Ž- _wandbruntime‚2649  _runtime‚2649ë ÛÝ'×O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2823054426013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974531307520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193959812453277 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884191492931836e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196257810194944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44761801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 trainer/global_step‚3399 _runtime‚2649.031011749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28596404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112913430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199090634016256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28596406 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2832823821312 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199091308381184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63889878 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49023552 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44762251 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49023552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49023554 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884191492931836e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191136758027264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2822399988261 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196257810194944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16165397 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199091308381184 M@DeviceStatsMonitor.on_train_batch_starüfLÏ™t/inactive_split.all.freed‚44761801 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214405481850368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 trainer/global_step‚3399 _runtime‚2649.031011749 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2833480183296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚214405505790464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28596404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2833480183296 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112913430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16165845 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974367683072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193959140976165 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196257828197888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199090634016256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564544 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191136740987904 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63909915 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196257828197888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217380037097984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217379849533440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 J¸2‘- _wandbruntime‚2649  _runtime‚2649’„ª·ç&’áM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2833643249152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217442373713920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112945900 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28604574 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2823216423225 trainer/global_step‚3399 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199148480861184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214467156528128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2834299624448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63908332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16170344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 _step‚276 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49037570 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196314163233792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214467132588032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975241125888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2833643249152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49037568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196314163233792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884191499895642e+09 _runtime‚2649.727731955 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196314181236736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112945900 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63928395 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199148480861184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191191645118464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63928395 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214467156528128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚191191662157824 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2834299624448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16169914 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63908332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16170344 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 _step‚276 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49037570 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217442552876544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28604572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196314163233792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44774918 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214467132588032 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63908332 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112965965 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2975241125888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2833643249152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49037568 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44774486 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196314163233792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 % _timestamp‚1.7884191499895642e+09 _runtime‚2649.727731955 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196314181236736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222528 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194015533019009 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112945900 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088”ÿT•g2”- _wandbruntime‚2649  _runtime‚2649ªW™Q•Ë 2loss/SRqqt1/assignment_loss‚1.6114391415349898 .loss/FRt4/detection_loss‚1.809102030005306 /loss/SRqqt1/detection_loss‚2.27600575743183 /loss/FBt1/detection_loss‚2.9497351429679175 epoch‚2 0loss/FRt3/assignment_loss‚3.1947187753610833 1loss/SRqqt2/assignment_loss‚2.044894838439567 /loss/FBt2/detection_loss‚2.8751075128714243 #loss/total_loss‚48.786228514206 _runtime‚2649.729183725 1loss/FBt2/assignment_loss‚0.45877694715489575 1loss/SRqqt3/detection_loss‚1.7092028175791105 0loss/FBt1/assignment_loss‚0.5885799865830976 /loss/FBt4/assignment_loss‚0.326860342174446 .loss/FRt1/detection_loss‚2.535635025964843 1loss/SRqqt4/detection_loss‚1.1410221411358743 .loss/FBt3/detection_loss‚2.562030702829361 2loss/SRqqt3/assignment_loss‚1.3118863453467686 /loss/FRt3/detection_loss‚2.4042976269888325 /loss/FRt4/assignment_loss‚1.903739043821891 trainer/global_step‚3399 /loss/FRt2/assignment_loss‚4.329711829914766 0loss/FBt3/assignment_loss‚1.1706546619534492 .loss/FRt2/detection_loss‚2.776242370114607 % _timestamp‚1.7884191499917357e+09 _step‚277 /loss/FRt1/assignment_loss‚4.406167434321509 1loss/SRqqt4/assignment_loss‚1.133669135659667 0loss/SRqqt2/detection_loss‚2.507030506219183 /loss/FBt4/detection_loss‚0.7597183978315826~ [wV–Ð 2loss/SRqqt1/assignment_loss‚1.6114391415349898 .loss/FRt4/detection_loss‚1.809102030005306 /loss/SRqqt1/detection_loss‚2.27600575743183 /loss/FBt1/detection_loss‚2.9497351429679175 epoch‚2 0loss/FRt3/assignment_loss‚3.1947187753610833 1loss/SRqqt2/assignment_loss‚2.044894838439567 /loss/FBt2/detection_loss‚2.8751075128714243 #loss/total_loss‚48.786228514206 _runtime‚2649.729183725 1loss/FBt2/assignment_loss‚0.45877694715489575 1loss/SRqqt3/detection_loss‚1.7092028175791105 0loss/FBt1/assignment_loss‚0.5885799865830976 /loss/FBt4/assignment_loss‚0.326860342174446 .loss/FRt1/detection_loss‚2.535635025964843 1loss/SRqqt4/detection_loss‚1.1410221411358743 .loss/FBt3/detection_loss‚2.562030702829361 2loss/SRqqt3/assignment_loss‚1.3118863453467686 /loss/FRt3/detection_loss‚2.4042976269888325 /loss/FRt4/assignment_loss‚1.903739043821891 trainer/global_step‚3399 /loss/FRt2/assignment_loss‚4.329711829914766 0loss/FBt3/assignment_loss‚1.1706546619534492 .loss/FRt2/detection_loss‚2.776242370114607 % _timestamp‚1.7884191499917357e+09 _step‚277 /loss/FRt1/assignment_loss‚4.406167434321509 1loss/SRqqt4/assignment_loss‚1.133669135659667 0loss/SRqqt2/detection_loss‚2.507030506219183 /loss/FBt4/detection_loss‚0.7597183978315826•žïùP’—‚(:‡ Ó¸äÔÚǶ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜eû€y˜‚(:î Ó¸äÔ¤Àç€ disk.rbd0.in‚0 disk.rbd0.out‚0% disk./.usageGB‚255.48444366455078 disk.sda3.out‚ 231.16015625 disk.sda2.out‚0' proc.memory.availableMB‚ 475.5859375 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 4.33203125& disk.nvme0c0n1.out‚10731.931640625 disk.sda2.in‚0 proc.cpu.threads‚17! disk.dm-0.out‚10731.931640625 disk.dm-1.in‚ 126.77734375 network.recv‚ 1.042166e+06 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125$ memory_percent‚97.09725379943848 disk.drbd2080.out‚0 disk.sda3.in‚ 126.77734375 disk.sda.out‚ 231.16015625! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 disk.drbd2266.out‚0$ disk.nvme0n1.out‚10731.931640625 cpu‚25.279265062677325 network.sent‚ 7.00668e+06) disk./.usagePercent‚17.14577538005445 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 231.16015625 disk.rbd1.out‚ 1.4072265625M~D’™‚(:‡ â¸äÔßÓ¤~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚177.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5172 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zSeŒyš‚(:î â¸äÔôå  network.sent‚ 7.115295e+06 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.4072265625 disk.sda.in‚ 126.77734375 proc.cpu.threads‚17$ disk.nvme0n1.out‚10775.455078125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.916370391845703 disk.nvme0c0n1.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0$ memory_percent‚97.09808826446533 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda.out‚ 231.6953125 disk.sda1.in‚0 disk.drbd2080.in‚0 network.recv‚ 1.048413e+06* disk./.usagePercent‚17.146091293351702% disk./.usageGB‚255.48915100097656& disk.nvme0c0n1.out‚10775.455078125 disk.sda3.out‚ 231.6953125 disk.drbd2266.out‚0 cpu‚25.277980951700798 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 231.6953125( proc.memory.availableMB‚ 475.44921875 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚ 4.33203125! disk.dm-0.out‚10775.455078125 disk.sda1.out‚0! proc.memory.rssMB‚ 2771.578125–V1R›‚(:’ ñ¸äÔ£³ü} gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.611' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.322200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)lyœ‚(:î ñ¸äÔàˆ€ network.recv‚ 1.052623e+06* disk./.usagePercent‚17.146240545744163 disk.sda.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.47021484375 disk.sda1.out‚0 disk.drbd2080.in‚0% disk./.usageGB‚255.49137496948242& disk.nvme0c0n1.out‚10820.052734375 disk.drbd2265.out‚0! disk.dm-0.out‚10820.052734375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.916370391845703 cpu‚25.27665160402349 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 232.3203125 disk.drbd1716.in‚0$ disk.nvme0n1.out‚10820.052734375 disk.rbd0.out‚0 network.sent‚ 7.121187e+06 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375! proc.memory.rssMB‚ 2771.578125 disk.sda.out‚ 232.3203125 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚97.09813594818115 disk.sda2.out‚0 disk.sda3.out‚ 232.3203125 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0( proc.memory.availableMB‚ 475.44140625ƒ9´>2- _wandbruntime‚2686  _runtime‚2686¿Èɞà trainer/global_step‚3449 % _timestamp‚1.7884191866625972e+09 _runtime‚2686.400137762 _step‚278 ' lr-AdamW/pg1‚0.00015165373375949193 ' lr-AdamW/pg2‚0.00015165373375949193ÛÎŸÈ _runtime‚2686.400137762 _step‚278 ' lr-AdamW/pg1‚0.00015165373375949193 ' lr-AdamW/pg2‚0.00015165373375949193 trainer/global_step‚3449 % _timestamp‚1.7884191866625972e+09–ǯI22 - _wandbruntime‚2686  _runtime‚2686ià°QÝ'¡×O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚196745854320685 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2863235327669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 _runtime‚2686.401122471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193881964555264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚3449 % _timestamp‚1.7884191866633399e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217488042673664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚199075462137344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199075480140288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114560321 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0W,߯â'¢ÜO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2873809627136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64815930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64815930 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2874465989120 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201949271764480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2874465989120 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚199075462137344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201949271764480 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45397176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199075480140288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114560321 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49724352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚196745182843573 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45397612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2863889765421 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64835967 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16392806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚196745854320685 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2863235327669 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 _runtime‚2686.401122471 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3017320181248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193881964555264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚3449 % _timestamp‚1.7884191866633399e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217488042673664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 J‘ôûç&¤áM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114592817 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199131815176192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196801574886417 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49738370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193936868685824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2864706200593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220567844966400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2687.119344575 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64834384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884191873811753e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2874629054976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199131833179136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193936851646464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64834384 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2864051762633 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114592817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114572752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884191873811753e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2874629054976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16397330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.smallÔCõþ_pool.peak‚ 258008576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217549669471232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49738368 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3018339106304 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45410304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚199131833179136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202006444231168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚196800903409097 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193936851646464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64854447 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199131833179136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49738368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚199131815176192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16396899 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114572752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚196801574886417 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2875285430272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49738370 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 _step‚280 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193936868685824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202007118609408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45409871 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2864706200593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220567844966400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64834384 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49738370 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚2687.119344575 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0˜ÿJµ2¦- _wandbruntime‚2687  _runtime‚2687<ï´†W§Ñ 1loss/SRqqt4/detection_loss‚0.9310267278376747 /loss/FBt3/detection_loss‚2.7237792313098907 trainer/global_step‚3449 0loss/FRt4/assignment_loss‚1.4815698793301215 %loss/total_loss‚45.89312853863014 1loss/SRqqt1/assignment_loss‚1.676232931986451 1loss/SRqqt3/assignment_loss‚1.290979296835356 0loss/FBt4/assignment_loss‚0.2649523652714845 /loss/FRt4/detection_loss‚1.9250915991858795 1loss/SRqqt2/detection_loss‚2.5162231343633987 1loss/SRqqt3/detection_loss‚2.0498154542820397 0loss/FBt2/assignment_loss‚0.7495563317622457 .loss/FBt2/detection_loss‚2.492367250578744 % _timestamp‚1.7884191873832002e+09 _runtime‚2687.120684424 .loss/FRt1/assignment_loss‚3.61083470582962 /loss/FRt2/assignment_loss‚4.150208983156416 1loss/FBt1/assignment_loss‚0.38888976722884333 /loss/FBt1/detection_loss‚2.4787423031197653 /loss/FBt4/detection_loss‚0.6909364405105651 epoch‚2 _step‚281 1loss/SRqqt2/assignment_loss‚2.055564602069995 2loss/SRqqt4/assignment_loss‚1.1399513779975035 .loss/FRt2/detection_loss‚3.097435203967271 0loss/FRt3/assignment_loss‚3.5491272781206216 1loss/FBt3/assignment_loss‚0.21664736489765346 /loss/FRt1/detection_loss‚2.3438460329174995 /loss/FRt3/detection_loss‚2.2695957506480426 /loss/SRqqt1/detection_loss‚1.79975452542305¡Ed\¨Ö /loss/FRt4/detection_loss‚1.9250915991858795 1loss/SRqqt2/detection_loss‚2.5162231343633987 1loss/SRqqt3/detection_loss‚2.0498154542820397 0loss/FBt2/assignment_loss‚0.7495563317622457 .loss/FBt2/detection_loss‚2.492367250578744 % _timestamp‚1.7884191873832002e+09 _runtime‚2687.120684424 .loss/FRt1/assignment_loss‚3.61083470582962 /loss/FRt2/assignment_loss‚4.150208983156416 1loss/FBt1/assignment_loss‚0.38888976722884333 /loss/FBt1/detection_loss‚2.4787423031197653 /loss/FBt4/detection_loss‚0.6909364405105651 epoch‚2 _step‚281 1loss/SRqqt2/assignment_loss‚2.055564602069995 2loss/SRqqt4/assignment_loss‚1.1399513779975035 .loss/FRt2/detection_loss‚3.097435203967271 0loss/FRt3/assignment_loss‚3.5491272781206216 1loss/FBt3/assignment_loss‚0.21664736489765346 /loss/FRt1/detection_loss‚2.3438460329174995 /loss/FRt3/detection_loss‚2.2695957506480426 /loss/SRqqt1/detection_loss‚1.79975452542305 1loss/SRqqt4/detection_loss‚0.9310267278376747 /loss/FBt3/detection_loss‚2.7237792313098907 trainer/global_step‚3449 0loss/FRt4/assignment_loss‚1.4815698793301215 %loss/total_loss‚45.89312853863014 1loss/SRqqt1/assignment_loss‚1.676232931986451 1loss/SRqqt3/assignment_loss‚1.290979296835356 0loss/FBt4/assignment_loss‚0.2649523652714845™±}î’©‚(:‡ €¹äﱂ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d0ÿpkª‚(:à €¹äÔßëˆ disk.sda3.out‚232.75 disk.rbd1.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17% disk./.usageGB‚255.49569702148438 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 cpu‚25.275475471506265 network.sent‚ 7.127632e+06 disk.rbd0.out‚0 disk.drbd2080.out‚0$ disk.nvme0n1.out‚10864.634765625 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚232.75 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0( proc.memory.availableMB‚ 475.37890625 disk.nvme0n1.in‚ 4.33203125& disk.nvme0c0n1.out‚10864.634765625 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚232.75 disk.sda2.out‚0 disk.drbd2265.out‚0! disk.dm-0.out‚10864.634765625 disk.rbd1.out‚ 1.47021484375 disk.sda1.out‚0* disk./.usagePercent‚17.146530602280457$ memory_percent‚97.09851741790771 network.recv‚ 1.057362e+06 disk.nvme0c0n1.in‚ 4.33203125¯ê©¢‘«‚(:† ¹äÔ鿳~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.991 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Imöt¬‚(:é ¹äÔÕÜÖ€" disk.dm-0.out‚10910.8154296875# proc.memory.availableMB‚475.125 cpu‚25.2742492617845 network.recv‚ 1.062522e+06* disk./.usagePercent‚17.146949430606476 disk.sda3.out‚ 233.2421875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2771.578125 proc.cpu.threads‚17 disk.drbd1716.in‚0% disk.nvme0n1.out‚10910.8154296875 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.916370391845703 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚10910.8154296875$ memory_percent‚97.10006713867188 network.sent‚ 7.235532e+06% disk./.usageGB‚255.50193786621094 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 233.2421875 disk.sda.out‚ 233.2421875 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0£‘Ö:2­- _wandbruntime‚2723  _runtime‚2723ÿWÉÇ®Á & lr-AdamW/pg1‚0.0001509130403848174 & lr-AdamW/pg2‚0.0001509130403848174 trainer/global_step‚3499 % _timestamp‚1.7884192237234073e+09 _runtime‚2723.460971249 _step‚282Aæö;Ì¯Æ _runtime‚2723.460971249 _step‚282 & lr-AdamW/pg1‚0.0001509130403848174 & lr-AdamW/pg2‚0.0001509130403848174 trainer/global_step‚3499 % _timestamp‚1.7884192237234073e+09š\¹%Ü2°- _wandbruntime‚2723  _runtime‚2723Ä·ïÑÝ'±×O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2914795432960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3060653607424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50425154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65762019 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3060489982976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2723.462171324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65741982 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50425152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¬O’Nå²ÜO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201893114079744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29413206 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204807909512704 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674364928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16620634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163624448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3060489982976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220570579556864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2723.462171324 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65741982 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223631233164288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚3499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50425152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196627171083264 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65762019 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201893132082688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2915451794944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116167134 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656361984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204807909512704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204808583877632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2914795432960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3060653607424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50425154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2915451794944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116187173 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50425152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16620211 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65762019 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204808583877632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430›ýœé2³- _wandbruntime‚2724  _runtime‚2724¹TGå&´ßM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116199604 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚201949485121536 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 # _timestamp‚1.78841922442341e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2904887102041 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚204865756357632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220632206354432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 # _timestamp‚1.78841922442341e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65780499 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46046029 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2905541540001 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50439170 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196682058174464 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949485121536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2915614860800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2916271236096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚201949467118592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29421374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50439170 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65760436 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163611136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29421372 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16625104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187551232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2904887102041 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚204865756357632 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2916271236096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116219669 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199586945276505 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50439168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46046478 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 _step‚287 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocat{‹S÷]ion.large_pool.freed‚51125952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29821606 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204710766022144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16848494 trainer/global_step‚3549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3103578897920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚2760.885010233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46670100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2955781238784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚223653116440064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207667221625856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51125954 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202317938055501 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202317266578389 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207666547260928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656361984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚223653092499968 _step‚287 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199372377611264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51125952 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853517 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66688071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199372360571904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29821606 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚204710766022144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956437600768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16848494 trainer/global_step‚3549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207667221625856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226756671397888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16848042 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51125954 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955585536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204710784025088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3103578897920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚2760.885010233 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674364928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46670100 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 258008576 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117793986 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66688071 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2955781238784 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚223653116440064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207667221625856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51125954 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202317938055501 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202317266578389 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚204710766022144 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 trainer/global_step‚3549 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29829774 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 _step‚288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3104597976576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 _runtime‚2761.600328261 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955585536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚223714743237632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51139970 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957257041920 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51139968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117846521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29829772 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51139970 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957257041920 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202372987143913 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202231296 trainer/global_step‚3549 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957257041920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207724394105856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66686488 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117826456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46682327 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29829774 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚461 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199427281741824 _step‚288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207724394105856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚226819162923008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946376879409 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204767137063936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204767137063936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46682790 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207723719727616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66706551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ¬JR2È- _wandbruntime‚2761  _runtime‚2761€Ž¹YÉÓ %loss/total_loss‚48.29632648390293 .loss/FRt2/assignment_loss‚4.24692428963525 0loss/FBt1/assignment_loss‚0.6381530930452487 0loss/FBt2/assignment_loss‚1.0524307981443901 /loss/FRt2/detection_loss‚2.9813765564135144 /loss/FBt4/detection_loss‚0.6343497490093936 trainer/global_step‚3549 /loss/FRt1/assignment_loss‚4.409146307092724 1loss/SRqqt1/assignment_loss‚1.588833299042149 2loss/SRqqt3/assignment_loss‚1.8472628805234477 1loss/SRqqt3/detection_loss‚1.8430056936211057 0loss/FRt3/assignment_loss‚3.1467863939025187 1loss/FBt3/assignment_loss‚0.34972637966275216 0loss/FBt4/assignment_loss‚0.2257711466629661 /loss/FRt4/detection_loss‚1.7378419679150265 /loss/FBt2/detection_loss‚3.3024886647860208 % _timestamp‚1.7884192618642385e+09 .loss/FBt1/detection_loss‚2.237382611807655 epoch‚2 _runtime‚2761.601695902 0loss/FRt4/assignment_loss‚1.3597190582527305 /loss/FBt3/detection_loss‚2.9299950182437895 .loss/FRt1/detection_loss‚2.472287613334078 _step‚289 2loss/SRqqt2/assignment_loss‚2.0723777147563727 2loss/SRqqt4/assignment_loss‚1.1979900241891543 /loss/FRt3/detection_loss‚2.2828844092108986 0loss/SRqqt1/detection_loss‚2.260678144341165 1loss/SRqqt2/detection_loss‚2.4751920820893467 1loss/SRqqt4/detection_loss‚1.0037225882212322SÊ…¨^ÊØ 0loss/FRt4/assignment_loss‚1.3597190582527305 /loss/FBt3/detection_loss‚2.9299950182437895 .loss/FRt1/detection_loss‚2.472287613334078 _step‚289 2loss/SRqqt2/assignment_loss‚2.0723777147563727 2loss/SRqqt4/assignment_loss‚1.1979900241891543 /loss/FRt3/detection_loss‚2.2828844092108986 0loss/SRqqt1/detection_loss‚2.260678144341165 1loss/SRqqt2/detection_loss‚2.4751920820893467 1loss/SRqqt4/detection_loss‚1.0037225882212322 %loss/total_loss‚48.29632648390293 .loss/FRt2/assignment_loss‚4.24692428963525 0loss/FBt1/assignment_loss‚0.6381530930452487 0loss/FBt2/assignment_loss‚1.0524307981443901 /loss/FRt2/detection_loss‚2.9813765564135144 /loss/FBt4/detection_loss‚0.6343497490093936 trainer/global_step‚3549 /loss/FRt1/assignment_loss‚4.409146307092724 1loss/SRqqt1/assignment_loss‚1.588833299042149 2loss/SRqqt3/assignment_loss‚1.8472628805234477 1loss/SRqqt3/detection_loss‚1.8430056936211057 0loss/FRt3/assignment_loss‚3.1467863939025187 1loss/FBt3/assignment_loss‚0.34972637966275216 0loss/FBt4/assignment_loss‚0.2257711466629661 /loss/FRt4/detection_loss‚1.7378419679150265 /loss/FBt2/detection_loss‚3.3024886647860208 % _timestamp‚1.7884192618642385e+09 .loss/FBt1/detection_loss‚2.237382611807655 epoch‚2 _runtime‚2761.601695902¡vÈÍ’Ë‚(:‡ ˹äÔÖÎÎ~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gâZ2{Ì‚(:ð ˹äÔ•œ×‚$ disk./.usageGB‚255.5472183227539 disk.rbd0.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 235.53515625 disk.nvme0n1.in‚ 4.33203125$ memory_percent‚97.10309505462646! proc.memory.rssMB‚ 2771.578125 disk.drbd2080.in‚0 cpu‚25.269229696362476 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚11088.4189453125( proc.memory.availableMB‚ 474.62890625) disk./.usagePercent‚17.14998824003791 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.916370391845703 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 235.53515625" disk.dm-0.out‚11088.4189453125 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.out‚0 network.sent‚ 7.36177e+06 network.recv‚ 1.080972e+06 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚11088.4189453125 disk.sda.out‚ 235.53515625 disk.sda1.out‚0CBÍ‚(:’ Ú¹äÔ…„~ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.011' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.002199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜„úàz΂(:ï Ú¹äÔû…¶€% disk./.usageGB‚255.57187271118164 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 235.99609375 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 network.recv‚ 1.087109e+06 disk.sda.out‚ 235.99609375" disk.dm-0.out‚11164.7431640625( proc.memory.availableMB‚ 474.87890625 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 235.99609375 disk.rbd1.out‚ 1.533203125( disk./.usagePercent‚17.1516428167317 disk.drbd1716.out‚0$ memory_percent‚97.10156917572021 network.sent‚ 7.470744e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2771.578125* proc.memory.percent‚16.916370391845703 cpu‚25.2680154254886% disk.nvme0n1.out‚11164.7431640625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚11164.7431640625 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0æÍ2Ï- _wandbruntime‚2798  _runtime‚2798øûˆLÉÐà ' lr-AdamW/pg1‚0.00014939019178573926 ' lr-AdamW/pg2‚0.00014939019178573926 trainer/global_step‚3599 % _timestamp‚1.7884192983234155e+09 _runtime‚2798.060962703 _step‚290vÒ¹•ÎÑÈ trainer/global_step‚3599 % _timestamp‚1.7884192983234155e+09 _runtime‚2798.060962703 _step‚290 ' lr-AdamW/pg1‚0.00014939019178573926 ' lr-AdamW/pg2‚0.00014939019178573926¢4>ä2Ò- _wandbruntime‚2798  _runtime‚2798lBË1Ü'ÓÖO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47306072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51826752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30230004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3146581171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2798.061867411 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210525859374080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67614123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47306521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226735653323264ä"êÏá'ÔÛO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202117567099904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 $ _timestamp‚1.788419298324124e+09 _step‚291 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207528435967488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚229882402313728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51826754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207528417964544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚3599 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2997423406592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2997423406592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17076515 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30230004 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚210525859374080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3146581171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2798.061867411 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚210525859374080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955585536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚205103979922909 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17076068 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699021 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67614123 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47306521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2985741345893 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚226735653323264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚229882210554880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚210525185009152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47306072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202117584139264 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51826752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51826752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67632603 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119453308 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17080569 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147448773632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119453308 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2986557780857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67612540 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229944912251392 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚2798.756939815 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2998242847744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430àB)ì&×æM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51840770 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51840768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚229944912251392 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47318741 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚2798.756939815 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2998242847744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378240 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3147608190464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚202172488269824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119473373 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30238172 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚210583031854080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207584771003392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437960 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67612540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699021 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67632603 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119453308 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51840768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17080569 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3147448773632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375296 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119453308 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚202172471230464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2987212218817 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2986557780857 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853517 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 258008576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955585536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚229944728894464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207584789006336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67612540 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207584789006336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¤`€2Ø- _wandbruntime‚2798  _runtime‚2798§õ¢TÙÎ /loss/FBt1/detection_loss‚2.4165496582334693 /loss/FBt2/detection_loss‚3.4026204148928323 trainer/global_step‚3599 _step‚293 .loss/FRt2/detection_loss‚2.792503172860426 0loss/FRt4/assignment_loss‚2.4652747926504714 _runtime‚ 2798.75841017 /loss/FRt1/assignment_loss‚3.893921345472336 1loss/FBt4/assignment_loss‚0.25598453732661214 0loss/SRqqt1/detection_loss‚2.117015344566769 1loss/SRqqt2/detection_loss‚2.1246833267600036 0loss/FRt3/assignment_loss‚3.8643419158701993 epoch‚2 /loss/FRt4/detection_loss‚1.9406579013751901 1loss/SRqqt2/assignment_loss‚2.294246873883314 2loss/SRqqt4/assignment_loss‚1.2318889525532724 0loss/FBt1/assignment_loss‚0.5629012191837485 0loss/FBt3/assignment_loss‚0.2589519868294398 /loss/FRt3/detection_loss‚2.0988719846521104 1loss/SRqqt3/detection_loss‚1.7958700238512113 %loss/total_loss‚49.32344854223204 /loss/FRt1/detection_loss‚2.2281135754151777 /loss/FRt2/assignment_loss‚4.217127284582923 0loss/FBt2/assignment_loss‚0.9268599195541659 /loss/SRqqt4/detection_loss‚1.20231839949886 -loss/FBt3/detection_loss‚3.18807460864385 /loss/FBt4/detection_loss‚0.6615471770482939 $ _timestamp‚1.788419299020969e+09 1loss/SRqqt1/assignment_loss‚1.770135412613551 2loss/SRqqt3/assignment_loss‚1.6129887139138122ú”•YÚÓ /loss/FRt3/detection_loss‚2.0988719846521104 1loss/SRqqt3/detection_loss‚1.7958700238512113 %loss/total_loss‚49.32344854223204 /loss/FRt1/detection_loss‚2.2281135754151777 /loss/FRt2/assignment_loss‚4.217127284582923 0loss/FBt2/assignment_loss‚0.9268599195541659 /loss/SRqqt4/detection_loss‚1.20231839949886 -loss/FBt3/detection_loss‚3.18807460864385 /loss/FBt4/detection_loss‚0.6615471770482939 $ _timestamp‚1.788419299020969e+09 1loss/SRqqt1/assignment_loss‚1.770135412613551 2loss/SRqqt3/assignment_loss‚1.6129887139138122 /loss/FBt1/detection_loss‚2.4165496582334693 /loss/FBt2/detection_loss‚3.4026204148928323 trainer/global_step‚3599 _step‚293 .loss/FRt2/detection_loss‚2.792503172860426 0loss/FRt4/assignment_loss‚2.4652747926504714 _runtime‚ 2798.75841017 /loss/FRt1/assignment_loss‚3.893921345472336 1loss/FBt4/assignment_loss‚0.25598453732661214 0loss/SRqqt1/detection_loss‚2.117015344566769 1loss/SRqqt2/detection_loss‚2.1246833267600036 0loss/FRt3/assignment_loss‚3.8643419158701993 epoch‚2 /loss/FRt4/detection_loss‚1.9406579013751901 1loss/SRqqt2/assignment_loss‚2.294246873883314 2loss/SRqqt4/assignment_loss‚1.2318889525532724 0loss/FBt1/assignment_loss‚0.5629012191837485 0loss/FBt3/assignment_loss‚0.2589519868294398¥xåQO’Û‚(:‡ é¹äÔÝÏ´~ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷yMuÜ‚(:ê é¹äÔ¯ÅÓ€ disk.drbd1716.out‚0' proc.memory.availableMB‚ 474.8359375 network.sent‚ 7.475656e+06 disk.rbd0.in‚0 cpu‚25.266856000349176 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0$ memory_percent‚97.10183143615723 disk.dm-0.in‚ 4.33203125) disk./.usagePercent‚17.15253986713168$ disk./.usageGB‚255.5852394104004 disk.sda2.in‚0 disk.dm-1.out‚ 236.59765625 disk.sda.out‚ 236.59765625! proc.memory.rssMB‚ 2771.578125 disk.dm-0.out‚11373.53515625 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.33203125# disk.nvme0n1.out‚11373.53515625% disk.nvme0c0n1.out‚11373.53515625 disk.drbd2080.out‚0 disk.sda3.out‚ 236.59765625 disk.rbd1.out‚ 1.533203125 disk.sda1.in‚0* proc.memory.percent‚16.916370391845703 network.recv‚ 1.090457e+06 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0¡*<œÝ‚(:‘ ø¹äÔ“¤ÿ} gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁwrÉvÞ‚(:ë ø¹äÔ®•­€* disk./.usagePercent‚17.153648635762476 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0% disk.nvme0c0n1.out‚11417.20703125( proc.memory.availableMB‚ 474.83203125* proc.memory.percent‚16.916370391845703 network.sent‚ 7.584705e+06 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 237.05078125 disk.nvme0n1.in‚ 4.33203125$ memory_percent‚97.10185527801514! proc.memory.rssMB‚ 2771.578125 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 237.05078125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 cpu‚25.26738769380344# disk.nvme0n1.out‚11417.20703125$ disk./.usageGB‚255.6017608642578 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 237.05078125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚11417.20703125 disk.rbd1.out‚ 1.533203125 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.sda1.out‚0 network.recv‚ 1.097181e+06ZVûó’ß‚(:‡ ‡ºäÔ׌~ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚185.082' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0164 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÛxà‚(:í ‡ºäÔ´·¨€ disk.rbd0.in‚0' disk.nvme0c0n1.out‚11477.4501953125 cpu‚25.292199844531492 disk.sda.out‚ 237.6171875 disk.drbd1716.in‚0* proc.memory.percent‚16.916370391845703$ disk./.usageGB‚255.6063461303711" disk.dm-0.out‚11477.4501953125 disk.rbd1.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2771.578125* disk./.usagePercent‚17.153956356818632 disk.sda2.out‚0$ memory_percent‚97.10187911987305 proc.cpu.threads‚17 network.recv‚ 1.100685e+06 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda3.out‚ 237.6171875 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.533203125% disk.nvme0n1.out‚11477.4501953125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-1.out‚ 237.6171875 disk.drbd2265.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 474.828125 network.sent‚ 7.589981e+067´µ¿’á‚(:‡ –ºäÔê­£~ gpu.0.gpu‚10.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€„šç{â‚(:ð –ºäÔÚŠ¿% disk./.usageGB‚255.61742401123047 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.578125 network.sent‚ 7.596054e+06 network.recv‚ 1.105353e+06 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 238.12890625 disk.drbd2266.out‚0& proc.memory.availableMB‚ 474.828125* disk./.usagePercent‚17.154699802698065 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚11525.8330078125 disk.sda3.out‚ 238.12890625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.916370391845703 proc.cpu.threads‚17 disk.sda2.out‚0" disk.dm-0.out‚11525.8330078125 disk.dm-1.in‚ 126.77734375$ memory_percent‚97.10187911987305 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 cpu‚25.31803490022702 disk.sda.out‚ 238.12890625 disk.sda1.in‚0% disk.nvme0n1.out‚11525.8330078125 disk.sda3.in‚ 126.77734375ïKÒî‘ã‚(:† ¥ºäÔÄã~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.928' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸàÚ{ä‚(:𠥺äÔÖÐÓ€ proc.cpu.threads‚17 network.sent‚ 7.601984e+06 disk.sda2.in‚0* proc.memory.percent‚16.916370391845703 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.rbd0.out‚0) disk./.usagePercent‚17.15499113927203 disk.sda2.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0( proc.memory.availableMB‚ 474.57421875 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0" disk.dm-0.out‚11578.1025390625 disk.dm-1.out‚ 238.76953125 disk.rbd1.out‚ 1.533203125# memory_percent‚97.1034288406372! proc.memory.rssMB‚ 2771.578125 disk.rbd1.in‚0 network.recv‚ 1.109491e+06% disk./.usageGB‚255.62176513671875% disk.nvme0n1.out‚11578.1025390625' disk.nvme0c0n1.out‚11578.1025390625 disk.sda3.out‚ 238.76953125 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 238.76953125 disk.sda1.out‚0 cpu‚25.34130365791369 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0Ø=în2å- _wandbruntime‚2866  _runtime‚2866‡›ª æ½T OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚68489106 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚213949270855168 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚210910992466432 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚30777404 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚233506589528064 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853517 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 121237802 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚213949270855168 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚210910974463488 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815168 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚176 _runtime‚2866.916538338 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815168 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚68473130 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158158848 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699021 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚230317760979456 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚48197342 ZLD/ñ%0¦eviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492829696 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492829696 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚233506747686912 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510832640 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953748 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914388 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚30777406 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17420114 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3188962767360 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚294 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230317784919552 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚178 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3026602812437 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3027093726825 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚205384276970496 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3038278388736 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚52748694 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚213948760022528 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚205384294009856 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3188828548608 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚205384276970496 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 3038278388736 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚52748694 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚213948760022528 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3037785559040 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24127 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚213948760022528 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚205384294009856 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914388 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚52748696 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚30777406 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17420114 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3188962767360 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚294 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚230317784919552 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24127 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 3037785559040 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17419938 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134218752 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚178 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3026602812437 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3027093726825 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955585536 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121221824 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚52748694 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953748 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚213985955177984 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3038150685696 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 158158848 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚210947804492288 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 121255910 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48208335 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121239932 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚205419587769344 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 _runtime‚2867.859106241 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚210947822495232 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 trainer/global_step‚3626 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚179 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30783910 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚52759502 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚68480432 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788419368121182e+09 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815168 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230355636712960 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 26274508828vp êÌR PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 121239932 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 3038150685696 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚17424425 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699021 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 121255910 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚52759502 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚213985955177984 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚177 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955585536 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510832640 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 RLDeviceStatsMonitor.on_validati¥$A_âon_batch_end/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17424248 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚30783908 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚68480432 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚213986466010624 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815168 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚230355660653056 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚205419570729984 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚68496408 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48208156 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚233544830388224 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 134218752 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 3189193675264 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 3038643515392 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚210947804492288 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953748 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15978 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15976 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚213985955177984 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3038150685696 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 158158848 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3189327894016 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚210947804492288 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚68496408 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚210947822495232 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 121255910 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚52759500 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚48208335 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3026966421845 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 121239932 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚208446537151829 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚205419587769344 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 _runtime‚2867.859106241 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚52759500 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚210947822495232 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3027457336233 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853517 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 trainer/global_step‚3626 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚179 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24400 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚213986466010624 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚30783910 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚208447045105577 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510832640 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚52759502 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚68480432 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492829696 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 $ _timestamp‚1.788419368121182e+09 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815168 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚230355636712960 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 B&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13145893812179565 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3220231831073761 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1279383897781372 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14084266126155853 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9812448620796204 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924339056015015 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,particle/sensitivity‚0.28071004152297974 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5078352689743042 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9744687676429749 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4903799295425415 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9959776997566223 *jet/accuracy_2_of_5‚0.4857463240623474 jet/accuracy_7_of_7‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.579582691192627 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4802417457103729 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12804467976093292 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4903799295425415 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9664682745933533 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32559528946876526 *jet/accuracy_1_of_3‚0.8398354053497314 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43908271193504333 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.125514417886734 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9893991351127625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9254925847053528 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12877018749713898 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1957070678472519 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9751443266868591 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14358355104923248 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12463218718767166 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143655061721802 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6629800200462341 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9833542704582214 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0064843944273889065 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46268147230148315 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.390695720911026 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4043771028518677 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 +jet/accuracy_3_of_6‚0.09615384787321091 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.510427713394165 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3552854359149933 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1302083432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 _runtime‚2880.497266469 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9703343510627747 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1611938178539276 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2573624849319458 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9766751527786255 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49907320737838745 +jet/accuracy_4_of_4‚0.00374779524281621 jet/accuracy_5_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977139830589294 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884193807575493e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23310324549674988 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07345513254404068 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5383424162864685 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2113095223903656 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027551021426916122 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18631979823112488 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22993826866149902 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4958670139312744 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15282480418682098 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07128409296274185 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1504252851009369 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5145075917243958 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.550000011920929 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01593715138733387 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2984545826911926 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0279953945428133 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.338925838470459 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12804467976093292 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014540152624249458 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9760299921035767 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5884584188461304 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1740044504404068 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981725215911865 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 +particle/specificity‚0.9419910311698914 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021785499528050423 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.542629063129425 jet/accuracy_6_of_6‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15758678317070007 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5167098045349121 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020619945600628853 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14529678225517273 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9943528771400452 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.992823600769043 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49923375248908997 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01697751320898533 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4796874523162842 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.522854208946228 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19828258454799652 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28616058826446533 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206268787384 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2670658826828003 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981725215911865 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5229167342185974 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3779069781303406 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39069071412086487 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3201720118522644 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_4‚0.8950937390327454 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10796476900577545 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4997679591178894 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3901747763156891 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24439577758312225 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1103239580988884 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5335842370986938 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.426454097032547 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39319923520088196 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10208161920309067 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚3626 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48510289192199707 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9908458590507507 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5420370101928711 @&Purity/3FRt*SRqqt2FBt/event_proportion‚ÃNÒÍ60.0003100198518950492 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8387380242347717 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012613188475370407 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,jet/accuracy_4_of_5‚0.013082437217235565 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5100950002670288 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9854236245155334 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.585611879825592 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3348214030265808 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4768518805503845 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12629297375679016 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.515740692615509 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33566805720329285 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4871668517589569 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.024298036471009254 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4428596794605255 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4708506762981415 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3583093285560608 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5290214419364929 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5751972794532776 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.1089477688074112 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000001788139343 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08757198601961136 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12507708370685577 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚296 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5359432697296143 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22235596179962158 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333432674408 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9724399447441101 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1702689677476883 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21029749512672424 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 epoch‚2 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1200508177280426 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 particle/accuracy_3_of_6‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10797284543514252 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5559868812561035 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 )loss/val_total_loss‚48.62959316807124 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4693754017353058 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44912973046302795 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.981271505355835 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929077327251434 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1235363557934761 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.020975174382328987 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9407994747161865 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15632173418998718 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14353099465370178 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9836218357086182 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45350003242492676 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011130008846521378 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_2‚0.7364082932472229 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2240978479385376 particle/accuracy_1_of_3‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40838780999183655 jet/accuracy_6_of_7‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4797776937484741 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5301748514175415 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15086208283901215 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4326741099357605 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_2‚0.0923030748963356 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9658913016319275 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06157763674855232 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827161192893982 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11893060803413391 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35501357913017273 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024420025292783976 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23888888955116272 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4860879182815552 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.483405739068985 jet/accuracy_4_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9702233672142029 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1836957484483719 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22116124629974365 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.059427399188280106 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.4743589758872986 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6345356106758118 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05337810888886452 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7validation_average_jet_accuracy‚0.41998645663261414 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4706800878047943 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5589079856872559 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435734391212463 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3093915581703186 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21210597455501556 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10357772558927536 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.993401825428009 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.97801274061203 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9877766966819763 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 particle/accuracy_4_of_5‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3971705138683319 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47284480929374695 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3853801190853119 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5788177847862244 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13601534068584442 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367:Ñߣ%I ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.993401825428009 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.97801274061203 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333134651184 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9877766966819763 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 particle/accuracy_4_of_5‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41999998688697815 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185185074806213 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3971705138683319 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.47284480929374695 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3853801190853119 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5788177847862244 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13601534068584442 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19158098101615906 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1734234243631363 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5017982721328735 +jet/accuracy_3_of_4‚0.04737471416592598 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4047555923461914 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16870759427547455 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9691043496131897 particle/accuracy_4_of_7‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35277774930000305 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9285714030265808 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2703055441379547 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9830503463745117 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.13450293242931366 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03304597735404968 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9625545144081116 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4057677686214447 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.47618618607521057 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05714285746216774 particle/accuracy_5_of_5‚1 6 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.236111119389534 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4656387269496918 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.19568149745464325 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.30243054032325745 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3208851218223572 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 (particle/f_score‚0.35200774669647217 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5324404835700989 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.10606061667203903 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16719402372837067 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4394223690032959 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4166666865348816 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 8"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.550000011920929 particle/accuracy_6_of_6‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9862968921661377 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26436781883239746 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333134651184 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13145893812179565 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3220231831073761 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1279383897781372 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14084266126155853 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9812448620796204 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924339056015015 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,particle/sensitivity‚0.28071004152297974 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5078352689743042 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9744687676429749 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_3_of_7‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4903799295425415 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9959776997566223 *jet/accuracy_2_of_5‚0.4857463240623474 jet/accuracy_7_of_7‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.579582691192627 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4802417457103729 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12804467976093292 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4903799295425415 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9664682745933533 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32559528946876526 *jet/accuracy_1_of_3‚0.8398354053497314 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43908271193504333 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 6 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.125514417886734 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9893991351127625 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9254925847053528 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.54365074634552 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12877018749713898 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1957070678472519 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9751443266868591 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14358355104923248 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12463218718767166 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8143655061721802 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6629800200462341 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9833542704582214 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0064843944273889065 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.46268147230148315 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.390695720911026 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4043771028518677 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 +jet/accuracy_3_of_6‚0.09615384787321091 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.510427713394165 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3552854359149933 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9912281036376953 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1302083432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 _runtime‚2880.497266469 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9703343510627747 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1611938178539276 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2573624849319458 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9766751527786255 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49907320737838745 +jet/accuracy_4_of_4‚0.00374779524281621 jet/accuracy_5_of_6‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9576389789581299 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977139830589294 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884193807575493e+09 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23310324549674988 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07345513254404068 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5383424162864685 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2113095223903656 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_3‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.027551021426916122 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18631979823112488 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22993826866149902 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4958670139312744 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15282480418682098 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07128409296274185 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1504252851009369 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5145075917243958 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.550000011920929 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01593715138733387 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/eveÒ4@3mJnt_proportion‚0 jet/accuracy_2_of_7‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2984545826911926 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0279953945428133 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.338925838470459 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12804467976093292 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014540152624249458 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4333333373069763 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9760299921035767 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5884584188461304 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1740044504404068 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981725215911865 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 +particle/specificity‚0.9419910311698914 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021785499528050423 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.542629063129425 jet/accuracy_6_of_6‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.15758678317070007 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5167098045349121 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020619945600628853 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14529678225517273 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9943528771400452 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.992823600769043 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.49923375248908997 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01697751320898533 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4796874523162842 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.522854208946228 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19828258454799652 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28616058826446533 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206268787384 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2670658826828003 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981725215911865 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5229167342185974 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3779069781303406 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39069071412086487 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3201720118522644 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_4‚0.8950937390327454 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.10796476900577545 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4997679591178894 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.0833333358168602 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3901747763156891 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24439577758312225 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1103239580988884 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5335842370986938 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.426454097032547 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39319923520088196 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10208161920309067 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚3626 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48510289192199707 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9908458590507507 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5420370101928711 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_4‚1 (particle/accuracy‚0.8387380242347717 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012613188475370407 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,jet/accuracy_4_of_5‚0.013082437217235565 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5100950002670288 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9854236245155334 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.585611879825592 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3348214030265808 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4768518805503845 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12629297375679016 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.515740692615509 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33566805720329285 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4871668517589569 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.024298036471009254 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4428596794605255 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4708506762981415 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3583093285560608 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5290214419364929 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5751972794532776 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9955357313156128 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004938271827995777 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.1089477688074112 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000001788139343 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08757198601961136 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12507708370685577 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚296 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5359432697296143 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22235596179962158 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4583333432674408 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9724399447441101 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4399999976158142 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1702689677476883 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21029749512672424 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 epoch‚2 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1200508177280426 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 particle/accuracy_3_of_6‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.10797284543514252 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5559868812561035 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 )loss/val_total_loss‚48.62959316807124 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4693754017353058 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44912973046302795 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.981271505355835 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929077327251434 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1235363557934761 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.020975174382328987 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9407994747161865 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15632173418998718 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.14353099465370178 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9836218357086182 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45350003242492676 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011130008846521378 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_2‚0.7364082932472229 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2240978479385376 particle/accuracy_1_of_3‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.40838780999183655 jet/accuracy_6_of_7‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4797776937484741 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5301748514175415 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15086208283901215 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4326741099357605 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_2_of_2‚0.0923030748963356 6 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.236111119389534 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9658913016319275 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06157763674855232 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827161192893982 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11893060803413391 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.35501357913017273 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024420025292783976 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.23888888955116272 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4860879182815552 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.483405739068985 jet/accuracy_4_of_7‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9702233672142029 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1836957484483719 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22116124629974365 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.059427399188280106 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.4743589758872986 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6345356106758118 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05337810888886452 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7validation_average_jet_accuracy‚0.41998645663261414 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4706800878047943 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5589079856872559 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435734391212463 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3093915581703186 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21210597455501556 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10357772558927536 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215¨múòð‚(:’ úäÔ§Ó§~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0By{ñ‚(:ð úäÔ‚ì‡) disk./.usagePercent‚17.18684359668221 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 239.7734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 cpu‚25.362610400676626 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125# proc.memory.rssMB‚ 2772.38671875$ disk./.usageGB‚256.0963897705078 disk.rbd1.out‚ 1.533203125 network.recv‚ 1.120088e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125* proc.memory.percent‚16.921305656433105 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 239.7734375# disk.dm-0.out‚12170.47314453125$ memory_percent‚93.68705749511719 network.sent‚ 7.716164e+06( disk.nvme0c0n1.out‚12170.47314453125 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 239.7734375% proc.memory.availableMB‚ 1034.3125 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125& disk.nvme0n1.out‚12170.47314453125Ù¿–%2ò- _wandbruntime‚2901  _runtime‚2901èõÉóà % _timestamp‚1.7884194018570006e+09 _runtime‚2901.594509062 _step‚297 ' lr-AdamW/pg1‚0.00014860835392070112 ' lr-AdamW/pg2‚0.00014860835392070112 trainer/global_step‚3649‚¼zÎôÈ ' lr-AdamW/pg1‚0.00014860835392070112 ' lr-AdamW/pg2‚0.00014860835392070112 trainer/global_step‚3649 % _timestamp‚1.7884194018570006e+09 _runtime‚2901.594509062 _step‚297©"~ü?2õ- _wandbruntime‚2901  _runtime‚2901S{csÝ'ö×O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53208332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212666379724288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31048156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68982099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580433 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2901.595462712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53208332 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210136453138142 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69002137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215727808906240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚298 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212666361721344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69002137 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3061429181952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53208330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215727134540800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122190429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3060772819456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232204337477120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212666361721344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48628589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680œ³H2d÷ÜO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53208332 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210136453138142 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69002137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68982099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215727808906240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232204361417216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚298 GADeviceStatsMonitor.on_}Mœô~"train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212666361721344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69002137 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3061429181952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215727808906240 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53208330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48629038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215727134540800 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 122190429 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3060772819456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122210469 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232204337477120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210135781661026 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215727134540800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212666361721344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48628589 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53208332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212666379724288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31048156 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚235417441272832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68982099 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17580433 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213079855616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3212903648768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚2901.595462712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944ªЖw2ø- _wandbruntime‚2902  _runtime‚2902Xå&ùßM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53222346 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48641639 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884194025638225e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235479935504384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210192173703874 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232266012154880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207141201645568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69000553 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 2902.3018088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 2902.3018088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884194025638225e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235479935504384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3213747155968 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53222348 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69020617 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122222899 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69000553 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48641180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210192173703874 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122242965 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212722732763136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210191502226550 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212722714760192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3062248623104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3062248623104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232265988214784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3050972058306 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235479735370752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚232266012154880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53222348 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17584856 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3213923349504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3050317620342 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69020617 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3061592247296 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚215784981386240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122242965 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207141201645568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207141184606208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31056324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212722732763136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69000553 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964«¤á‰=2û- _wandbruntime‚2902  _runtime‚2902É·ÖøYüÓ .loss/FRt1/detection_loss‚2.565562387307485 /loss/FRt2/detection_loss‚3.1149916211764017 0loss/SRqqt1/detection_loss‚2.308719729906634 1loss/SRqqt2/detection_loss‚2.3901487170038997 /loss/FRt1/assignment_loss‚4.305606667812054 /loss/FRt3/detection_loss‚2.1341672077775002 % _timestamp‚1.7884194025656555e+09 1loss/SRqqt4/assignment_loss‚1.262790395025636 /loss/FRt2/assignment_loss‚3.854508403937022 2loss/SRqqt1/assignment_loss‚1.6601587576105405 1loss/SRqqt3/detection_loss‚1.6961366023336137 0loss/SRqqt4/detection_loss‚0.931404815960518 /loss/FBt4/detection_loss‚0.8606766653288266 trainer/global_step‚3649 0loss/FBt1/assignment_loss‚0.5267774229421485 /loss/FBt2/detection_loss‚3.1472380995750426 0loss/FRt4/assignment_loss‚1.3591372590315969 2loss/SRqqt2/assignment_loss‚1.6013954209314811 0loss/FBt4/assignment_loss‚0.2193669322463911 /loss/FBt1/detection_loss‚2.8909491759080153 .loss/FBt3/detection_loss‚2.951979228428432 %loss/total_loss‚46.45675519759702 0loss/FRt3/assignment_loss‚2.9263902912537256 _step‚300 1loss/FBt2/assignment_loss‚0.47149285798698204 /loss/FRt4/detection_loss‚1.5872616106480884 _runtime‚2902.303148289 epoch‚3 2loss/SRqqt3/assignment_loss‚1.3560881848846162 0loss/FBt3/assignment_loss‚0.3338067425803664~²¢ë^ýØ 0loss/FBt1/assignment_loss‚0.5267774229421485 /loss/FBt2/detection_loss‚3.1472380995750426 0loss/FRt4/assignment_loss‚1.3591372590315969 2loss/SRqqt2/assignment_loss‚1.6013954209314811 0loss/FBt4/assignment_loss‚0.2193669322463911 /loss/FBt1/detection_loss‚2.8909491759080153 .loss/FBt3/detection_loss‚2.951979228428432 %loss/total_loss‚46.45675519759702 0loss/FRt3/assignment_loss‚2.9263902912537256 _step‚300 1loss/FBt2/assignment_loss‚0.47149285798698204 /loss/FRt4/detection_loss‚1.5872616106480884 _runtime‚2902.303148289 epoch‚3 2loss/SRqqt3/assignment_loss‚1.3560881848846162 0loss/FBt3/assignment_loss‚0.3338067425803664 .loss/FRt1/detection_loss‚2.565562387307485 /loss/FRt2/detection_loss‚3.1149916211764017 0loss/SRqqt1/detection_loss‚2.308719729906634 1loss/SRqqt2/detection_loss‚2.3901487170038997 /loss/FRt1/assignment_loss‚4.305606667812054 /loss/FRt3/detection_loss‚2.1341672077775002 % _timestamp‚1.7884194025656555e+09 1loss/SRqqt4/assignment_loss‚1.262790395025636 /loss/FRt2/assignment_loss‚3.854508403937022 2loss/SRqqt1/assignment_loss‚1.6601587576105405 1loss/SRqqt3/detection_loss‚1.6961366023336137 0loss/SRqqt4/detection_loss‚0.931404815960518 /loss/FBt4/detection_loss‚0.8606766653288266 trainer/global_step‚3649¬wÄüòœþ‚(:‘ ÒºäÔÌž·~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$¥¢½ÿ‚(:ç ÒºäÔ¾ÝÚ€ disk.drbd2266.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚12245.67236328125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.533203125# disk.dm-0.out‚12245.67236328125 disk.sda1.out‚0 network.sent‚ 7.840595e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 network.recv‚ 1.126731e+06$ disk./.usageGB‚256.1221466064453 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2772.38671875( disk./.usagePercent‚17.1885721595534 disk.sda3.out‚ 240.34375 disk.drbd2265.in‚0EÆ̬µ disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125 cpu‚25.361019688087154 disk.dm-1.out‚ 240.34375 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.921305656433105 disk.sda.out‚ 240.34375 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚12245.67236328125$ memory_percent‚93.68743896484375# proc.memory.availableMB‚1034.25…¨X’€‚(:‡ áºäÔÕ¥~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ˆvð}‚(:ò áºäÔ¸Ò˜€% disk./.usageGB‚256.14157485961914 disk.rbd1.in‚0 disk.sda.out‚ 240.85546875 disk.nvme0n1.in‚ 4.33203125$ disk.nvme0n1.out‚12763.083984375! disk.dm-0.out‚12763.083984375 disk.dm-1.out‚ 240.85546875 disk.rbd0.out‚0 disk.dm-0.in‚ 4.33203125# proc.memory.rssMB‚ 2772.38671875 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚12763.083984375 disk.sda2.in‚0 cpu‚25.35944795804245 proc.cpu.threads‚17 network.sent‚ 7.948824e+06 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1034.20703125* disk./.usagePercent‚17.189876005925285 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.68770122528076 disk.rbd0.in‚0 disk.sda3.out‚ 240.85546875* proc.memory.percent‚16.921305656433105 network.recv‚ 1.132757e+06 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 1.533203125 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0L¤ 92‚- _wandbruntime‚2938  _runtime‚2938ÊDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2938.708073447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238542785055744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚218586446654464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31456558 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212921823528434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53909130 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218586446654464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3255886755328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3101758625280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3090990962542 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚218585772289024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49260887 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31456558 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212921823528434 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215484031666688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3102414987776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69928189 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31456556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚302 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53909130 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53909130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚238542595394560 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚218586446654464 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3255886755328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3101758625280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3090990962542 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚2938.708073447 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928®åšKO2ˆ- _wandbruntime‚2939  _runtime‚2939èË,&‰áM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17809275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 trainer/global_step‚3699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212977544093958 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53923146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2939.414726795 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884194396765673e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3102578053120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53923146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238605295015936 SEDeviceStatsMonitor.on_train_batczb„äh_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209886391134208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928ëe´ì&ŠæM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212978215571282 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53923148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚218643619134464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69926605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49273570 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚218643619134464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3256584464384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3102578053120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53923146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3091152959750 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123869817 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69926605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123849751 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _step‚303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238605295015936 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209886391134208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17809275 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235348549038080 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3091807397714 trainer/global_step‚3699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212977544093958 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215540366702592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53923146 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53923148 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69946669 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3103234428928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚2939.414726795 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3103234428928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238605109562368 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123869817 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123849751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884194396765673e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18030219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49895175 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221445084402688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31864956 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18030655 _step‚306 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49895613 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31864958 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212576710571008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884194765086799e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3131171864194 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3143400793600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70854241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54609930 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238369411243520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238369435183616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884194765086799e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3131171864194 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3143400793600 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚241668098789888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241667902837248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70854241 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70834203 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚n.=54609930 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238369411243520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54609932 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221444410037248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218301683609088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3143400793600 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238369435183616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18030219 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49895175 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414²𲀈2š- _wandbruntime‚2976  _runtime‚2976-·MRæ&›àM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3132642737122 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18034642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238431061981184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3299346749952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3143563858944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70872721 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3144220234752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788419477232929e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49908209 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 _step‚307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54623948 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ðå¹Ië&œåM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚3749 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125496669 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125476603 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70872721 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3144220234752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241730598379008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚221502256882688 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241730408731136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788419477232929e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49908209 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3144220234752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221502256882688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125496669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70852657 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49907766 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3299512457728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚215763585961366 _step‚307 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218358036647936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54623948 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31873124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3132642737122 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54623946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18034642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218358018644992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3131988299158 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚238431061981184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3299346749952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221501582503936 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70852657 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3143563858944 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71760255 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215321900059648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221119317548544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215321917099008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 trainer/global_step‚3799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884195136829553e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚3013.420676798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3184386599424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55310730 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244793543762432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18257052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18256604 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55310730 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3184386599424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221119317548544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71760255 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚221119335551488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚218493907263250 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215321900059648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚218494578740366 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3172007203602 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50530410 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221119317548544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55310732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚241451972066816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71760255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215321917099008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224303722150912 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221119335551488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3183730236928 trainer/global_step‚3799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884195136829553e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244793343615488 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127070985 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71780293 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50529960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 _runtime‚3013.420676798 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224303722150912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127091025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224303047785472 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224303047785472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241451948126720 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50543081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244856036732416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244855838695936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224360220252160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚311 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221175688590336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788419514382434e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚3014.120548773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127103455 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244856036732416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244855838695936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241513598864384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127123521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71798773 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241513622804480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3173478076530 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚221175670587392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smÝÓæ’\all_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚224360220252160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚311 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221175688590336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55324746 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218550299306098 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3342239831552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32281526 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788419514382434e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚3014.120548773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3185206040576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18261555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3185206040576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127123521 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55324748 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55324746 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50543081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50542646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32281524·ÓP‰æ2­- _wandbruntime‚3014  _runtime‚3014&?òöU®Ï /loss/FRt2/detection_loss‚2.8497855867658344 /loss/FRt3/detection_loss‚2.2411253976502588 /loss/FBt2/detection_loss‚2.8553641709414395 $ _timestamp‚1.788419514384576e+09 1loss/SRqqt4/assignment_loss‚1.271259940539797 0loss/FBt3/assignment_loss‚0.4454977810382843 1loss/SRqqt4/detection_loss‚1.2209086300775644 /loss/FBt4/detection_loss‚0.7999459908415492 /loss/FRt4/assignment_loss‚2.020712262019515 0loss/SRqqt1/detection_loss‚2.043485040344843 0loss/SRqqt3/detection_loss‚1.968593467684353 /loss/FBt1/detection_loss‚2.5276654145934363 %loss/total_loss‚48.99784393385196 2loss/SRqqt1/assignment_loss‚1.6143199132709969 2loss/SRqqt2/assignment_loss‚2.0478116129007606 1loss/FBt4/assignment_loss‚0.23529148597460756 /loss/FRt1/assignment_loss‚3.763325910414419 /loss/FRt4/detection_loss‚1.7108022347092628 -loss/FRt1/detection_loss‚2.30206355356401 2loss/SRqqt3/assignment_loss‚1.8667852390630573 0loss/FBt1/assignment_loss‚0.6900943490591916 trainer/global_step‚3799 /loss/FRt2/assignment_loss‚4.916015754427229 /loss/FRt3/assignment_loss‚3.348193049430847 0loss/SRqqt2/detection_loss‚2.637844289342562 /loss/FBt3/detection_loss‚3.1681648015975954 epoch‚3 0loss/FBt2/assignment_loss‚0.4527880576005456 _runtime‚3014.122044766 _step‚312dôZ¯Ô /loss/FRt4/detection_loss‚1.7108022347092628 -loss/FRt1/detection_loss‚2.30206355356401 2loss/SRqqt3/assignment_loss‚1.8667852390630573 0loss/FBt1/assignment_loss‚0.6900943490591916 trainer/global_step‚3799 /loss/FRt2/assignment_loss‚4.916015754427229 /loss/FRt3/assignment_loss‚3.348193049430847 0loss/SRqqt2/detection_loss‚2.637844289342562 /loss/FBt3/detection_loss‚3.1681648015975954 epoch‚3 0loss/FBt2/assignment_loss‚0.4527880576005456 _runtime‚3014.122044766 _step‚312 /loss/FRt2/detection_loss‚2.8497855867658344 /loss/FRt3/detection_loss‚2.2411253976502588 /loss/FBt2/detection_loss‚2.8553641709414395 $ _timestamp‚1.788419514384576e+09 1loss/SRqqt4/assignment_loss‚1.271259940539797 0loss/FBt3/assignment_loss‚0.4454977810382843 1loss/SRqqt4/detection_loss‚1.2209086300775644 /loss/FBt4/detection_loss‚0.7999459908415492 /loss/FRt4/assignment_loss‚2.020712262019515 0loss/SRqqt1/detection_loss‚2.043485040344843 0loss/SRqqt3/detection_loss‚1.968593467684353 /loss/FBt1/detection_loss‚2.5276654145934363 %loss/total_loss‚48.99784393385196 2loss/SRqqt1/assignment_loss‚1.6143199132709969 2loss/SRqqt2/assignment_loss‚2.0478116129007606 1loss/FBt4/assignment_loss‚0.23529148597460756 /loss/FRt1/assignment_loss‚3.763325910414419¸%1¬’°‚(:‡ »»äÔ«‰»~ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ëûs±‚(:è »»äÔÌă disk.sda.in‚ 126.77734375( disk.nvme0c0n1.out‚13078.70458984375& proc.memory.availableMB‚ 1111.28125 disk.drbd2266.out‚0& disk.nvme0n1.out‚13078.70458984375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 244.34375 network.sent‚ 8.18932e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚13078.70458984375 disk.drbd2080.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda.out‚ 244.34375 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375) disk./.usagePercent‚17.19478802248475$ disk./.usageGB‚256.2147674560547 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚93.21727752685547 disk.rbd1.in‚0 disk.dm-1.in‚ 126.77734375" proc.memory.rssMB‚ 2693.1953125 cpu‚25.350193338799112 proc.cpu.threads‚17 network.recv‚ 1.16134e+06 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 244.34375* proc.memory.percent‚16.437959671020508¢UA’²‚(:‡ Ê»äÔ†•â gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lŒ“õy³‚(:î Ê»äÔÙϾ disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2693.1953125 proc.cpu.threads‚17* disk./.usagePercent‚17.194484397549143 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2266.in‚0* proc.memory.percent‚16.437959671020508 network.sent‚ 8.298065e+06 network.recv‚ 1.168241e+06% disk./.usageGB‚256.21024322509766 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0$ memory_percent‚93.22021007537842) proc.memory.availableMB‚ 1110.80078125 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚13126.49365234375 disk.sda2.out‚0# disk.dm-0.out‚13126.49365234375 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0( disk.nvme0c0n1.out‚13126.49365234375 disk.dm-1.out‚ 244.71875 cpu‚25.34871212996735 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.sda3.out‚ 244.71875 disk.drbd2080.out‚0 disk.rbd1.out‚ 1.533203125 disk.sda.out‚ 244.71875ur§èœ´‚(:‘ Ù»äÔÀ“¢~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.789999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݵ!ûsµ‚(:è Ù»äÔ¸ŒÄ€* disk./.usagePercent‚17.195562701283365 disk.dm-0.in‚ 4.33203125 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚13193.44775390625 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 network.recv‚ 1.172133e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 1.533203125 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.sda2.out‚0# disk.dm-0.out‚13193.44775390625 disk.rbd1.in‚0$ memory_percent‚93.22171211242676( proc.memory.availableMB‚ 1110.5546875" proc.memory.rssMB‚ 2693.1953125% disk./.usageGB‚256.22631072998047( disk.nvme0c0n1.out‚13193.44775390625 disk.sda2.in‚0 disk.dm-1.out‚245.375 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0* proc.memory.percent‚16.437959671020508 cpu‚25.347269521848926 network.sent‚ 8.304031e+06 disk.sda3.out‚245.375 disk.sda.out‚245.375;x]ª2¶- _wandbruntime‚3050  _runtime‚3050lƒO›É·Ã ' lr-AdamW/pg2‚0.00014534874605447274 trainer/global_step‚3849 % _timestamp‚1.7884195509066498e+09 _runtime‚3050.644230919 _step‚313 ' lr-AdamW/pg1‚0.00014534874605447274ÅÁGóÎ¸È trainer/global_step‚3849 % _timestamp‚1.7884195509066498e+09 _runtime‚3050.644230919 _step‚313 ' lr-AdamW/pg1‚0.00014534874605447274 ' lr-AdamW/pg2‚0.00014534874605447274¹ÞÔÐ2¹- _wandbruntime‚3050  _runtime‚3050½âŒªÝ'º×O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128697837 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244534508950016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3849 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56011532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72686307 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56011530 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247918856526336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚314 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 oÔ\»ÜO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚3849 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227161685533696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56011532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3225372405248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247918671059456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3213496980766 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3225372405248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72686307 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56011530 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227162359899136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51165921 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247918856526336 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚314 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223936969490944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218067123627008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72706345 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223936987493888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128717877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3384347576320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3384186049536 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.fð‡œ† reed‚ 128697837 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72706345 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32681756 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223936969490944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18483729 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244534508950016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚434 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221279949130658º’FÜ2¼- _wandbruntime‚3051  _runtime‚3051†Rså&½ßM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56025548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72704761 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218122027757568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788419551602259e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214313415938 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3226191846400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227219532379136 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51178592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223993322529792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56025546 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225535470592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223993340532736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72724825 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3226191846400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18488666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _runtime‚ 3051.34021449 _step‚315 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385207989760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56025548 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128730307 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227219532379136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218122010718208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3225535470592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18488220 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225535470592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223993340532736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244596135747584 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72724825 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3226191846400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18488666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 _runtime‚ 3051.34021449 _step‚315 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128750373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3385207989760 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56025548 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56025546 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227218858000384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32689926 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3213658977974 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72704761 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51178144 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218122027757568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 $ _timestamp‚1.788419551602259e+09 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128750373 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3214313415938 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3226191846400 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244596159687680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944»æ1‚–2¿- _wandbruntime‚3051  _runtime‚3051Õ3­<\ÀÖ 2loss/SRqqt2/assignment_loss‚2.0398621664541525 1loss/SRqqt3/assignment_loss‚1.934620362563202 /loss/FBt3/detection_loss‚3.9843570987383523 /loss/FBt4/detection_loss‚0.6821282077816138 1loss/FBt3/assignment_loss‚0.29309013029220904 /loss/FRt4/detection_loss‚1.7980654354172725 1loss/SRqqt1/detection_loss‚1.8458331943142647 _step‚316 0loss/FRt1/assignment_loss‚3.3488106930509525 /loss/FBt1/detection_loss‚2.6516563859250812 1loss/FBt4/assignment_loss‚0.18698839942080436 trainer/global_step‚3849 _runtime‚3051.341246739 /loss/FRt4/assignment_loss‚1.884260176508515 2loss/SRqqt4/assignment_loss‚1.1897318689596086 2loss/SRqqt1/assignment_loss‚1.7133278534108518 1loss/SRqqt3/detection_loss‚1.7721484377980232 1loss/SRqqt4/detection_loss‚1.2536353248925436 /loss/FRt1/detection_loss‚2.3767373498449933 /loss/FRt2/assignment_loss‚4.219933943314985 0loss/FBt1/assignment_loss‚0.4470385031567648 epoch‚3 /loss/FRt2/detection_loss‚2.9382696386539573 .loss/FRt3/detection_loss‚2.400931536200199 0loss/SRqqt2/detection_loss‚2.264148249858763 /loss/FBt2/detection_loss‚2.6154475013415017 % _timestamp‚1.7884195516037984e+09 &loss/total_loss‚47.976538109491585 0loss/FBt2/assignment_loss‚0.8316093186537424 /loss/FRt3/assignment_loss‚3.303906332939229øTîaÁÛ 1loss/FBt4/assignment_loss‚0.18698839942080436 trainer/global_step‚3849 _runtime‚3051.341246739 /loss/FRt4/assignment_loss‚1.884260176508515 2loss/SRqqt4/assignment_loss‚1.1897318689596086 2loss/SRqqt1/assignment_loss‚1.7133278534108518 1loss/SRqqt3/detection_loss‚1.7721484377980232 1loss/SRqqt4/detection_loss‚1.2536353248925436 /loss/FRt1/detection_loss‚2.3767373498449933 /loss/FRt2/assignment_loss‚4.219933943314985 0loss/FBt1/assignment_loss‚0.4470385031567648 epoch‚3 /loss/FRt2/detection_loss‚2.9382696386539573 .loss/FRt3/detection_loss‚2.400931536200199 0loss/SRqqt2/detection_loss‚2.264148249858763 /loss/FBt2/detection_loss‚2.6154475013415017 % _timestamp‚1.7884195516037984e+09 &loss/total_loss‚47.976538109491585 0loss/FBt2/assignment_loss‚0.8316093186537424 /loss/FRt3/assignment_loss‚3.303906332939229 2loss/SRqqt2/assignment_loss‚2.0398621664541525 1loss/SRqqt3/assignment_loss‚1.934620362563202 /loss/FBt3/detection_loss‚3.9843570987383523 /loss/FBt4/detection_loss‚0.6821282077816138 1loss/FBt3/assignment_loss‚0.29309013029220904 /loss/FRt4/detection_loss‚1.7980654354172725 1loss/SRqqt1/detection_loss‚1.8458331943142647 _step‚316 0loss/FRt1/assignment_loss‚3.3488106930509525 /loss/FBt1/detection_loss‚2.6516563859250812¼E™t¥x‚(:í è»äÔØ——€ disk.sda1.in‚0 disk.rbd1.out‚ 1.533203125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 245.96875 disk.drbd2080.out‚0 disk.drbd2266.out‚0& disk.nvme0n1.out‚13249.47509765625 disk.dm-0.in‚ 4.33203125 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.sda.out‚ 245.96875 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.437959671020508 cpu‚25.345749682541356 network.sent‚ 8.310331e+06 network.recv‚ 1.176935e+06 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0# disk.dm-0.out‚13249.47509765625" proc.memory.rssMB‚ 2693.1953125 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚13249.47509765625 disk.dm-1.out‚ 245.96875 disk.drbd1716.out‚0* disk./.usagePercent‚17.195786707875822$ disk./.usageGB‚256.2296485900879 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚93.21885108947754 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375( proc.memory.availableMB‚ 1111.0234375>7Õ”“Â(:ˆ è»äÔ›Þ”š gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍŒ„Q’Ä‚(:‡ ÷»äÔ›ê~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 `–|Å‚(:ñ ÷»äÔ㉠ disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125* proc.memory.percent‚16.437959671020508 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 246.49609375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.533203125" proc.memory.rssMB‚ 2693.1953125 network.sent‚ 8.418275e+06$ disk./.usageGB‚256.2519187927246 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚13301.66259765625# disk.dm-0.out‚13301.66259765625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 cpu‚25.344220405555358* disk./.usagePercent‚17.197281279860718 disk.sda2.in‚0 disk.dm-1.out‚ 246.49609375$ memory_percent‚95.25680541992188 disk.drbd1716.in‚0& disk.nvme0n1.out‚13301.66259765625 disk.rbd0.in‚0 disk.drbd2265.out‚0# proc.memory.availableMB‚777.125 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.sda.out‚ 246.49609375 network.recv‚ 1.182426e+06 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375ä¢ßµ2Æ- _wandbruntime‚3087  _runtime‚3087%ó"KÇÇÁ _runtime‚3087.652171695 _step‚317 & lr-AdamW/pg1‚0.0001445016291258362 & lr-AdamW/pg2‚0.0001445016291258362 trainer/global_step‚3899 % _timestamp‚1.7884195879146035e+09x«ùdÌÈÆ & lr-AdamW/pg1‚0.0001445016291258362 & lr-AdamW/pg2‚0.0001445016291258362 trainer/global_step‚3899 % _timestamp‚1.7884195879146035e+09 _runtime‚3087.652171695 _step‚317½R2É- _wandbruntime‚3087  _runtime‚3087<æ  Ê×O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73612359 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51801065 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73632397 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226754621433344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226754621433344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230020323281920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73632397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3265701848576 H>DeviceStaZ&ÒÓtsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130344729 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254332320174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3253677882418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73612359 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226754639436288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3265701848576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220812330155008 _runtime‚3087.653082254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33090158 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚230020323281920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73632397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3265701848576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230020323281920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56712332 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130344729 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3254332320174 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3253677882418 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3427197445632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226754639436288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18710909 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251044219338752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73612359 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226754639436288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247617045833216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3265701848576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33090156 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3427365263872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224065990998066 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220812330155008 _runtime‚3087.653082254 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33090158 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251044411097088 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51814195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3088.340436064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3255148755346 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73650877 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73650877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚319 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3266521276416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884195886023693e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251106915701760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880”0ÙéÎæM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226810974472192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230077495748608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220867217246208 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130377225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18715869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251106915701760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3428219130880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3267177652224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130377225 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚230078170127360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226810992475136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73630813 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51814195 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230078170127360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚3088.340436064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56726346 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3255148755346 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73630813 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226810974472192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3266521276416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51813733 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73650877 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73650877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18715409 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33098326 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚319 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224122383040914 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚3899 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3266521276416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56726348 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3267177652224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130357159 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251106717665280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884195886023693e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33098324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889¿Y@w2Ï- _wandbruntime‚3088  _runtime‚3088¶,®7VÐÐ 0loss/FRt3/assignment_loss‚3.3398035656322134 _step‚320 0loss/FBt1/assignment_loss‚0.2776524288856308 /loss/FRt3/detection_loss‚2.3467824079773645 .loss/FBt4/detection_loss‚0.599880376242153 _runtime‚3088.341609919 /loss/FRt2/assignment_loss‚4.430783596038818 0loss/FRt4/assignment_loss‚1.9580395072698593 /loss/FRt4/detection_loss‚1.7938765990906036 0loss/SRqqt2/detection_loss‚2.549178785747952 2loss/SRqqt1/assignment_loss‚1.6907695224508643 1loss/SRqqt3/assignment_loss‚1.708798181339507 0loss/FBt2/assignment_loss‚0.6981024715577304 0loss/FBt3/assignment_loss‚0.4540942057925956 0loss/FBt4/assignment_loss‚0.2549894058790633 1loss/SRqqt4/detection_loss‚1.2827719548949972 /loss/FRt1/assignment_loss‚4.363335915068363 2loss/SRqqt4/assignment_loss‚1.2616850400227122 /loss/FBt3/detection_loss‚3.0836374958356223 trainer/global_step‚3899 2loss/SRqqt2/assignment_loss‚2.2818228360265493 .loss/FRt1/detection_loss‚2.365347854000457 1loss/SRqqt1/detection_loss‚2.0495967927078405 %loss/total_loss‚48.97433945875315 % _timestamp‚1.7884195886041825e+09 epoch‚3 .loss/FRt2/detection_loss‚3.059922435283661 1loss/SRqqt3/detection_loss‚1.6599766633340292 .loss/FBt1/detection_loss‚2.808209026232362 .loss/FBt2/detection_loss‚2.655282391442193ô¬:[ÑÕ 1loss/SRqqt3/assignment_loss‚1.708798181339507 0loss/FBt2/assignment_loss‚0.6981024715577304 0loss/FBt3/assignment_loss‚0.4540942057925956 0loss/FBt4/assignment_loss‚0.2549894058790633 1loss/SRqqt4/detection_loss‚1.2827719548949972 /loss/FRt1/assignment_loss‚4.363335915068363 2loss/SRqqt4/assignment_loss‚1.2616850400227122 /loss/FBt3/detection_loss‚3.0836374958356223 trainer/global_step‚3899 2loss/SRqqt2/assignment_loss‚2.2818228360265493 .loss/FRt1/detection_loss‚2.365347854000457 1loss/SRqqt1/detection_loss‚2.0495967927078405 %loss/total_loss‚48.97433945875315 % _timestamp‚1.7884195886041825e+09 epoch‚3 .loss/FRt2/detection_loss‚3.059922435283661 1loss/SRqqt3/detection_loss‚1.6599766633340292 .loss/FBt1/detection_loss‚2.808209026232362 .loss/FBt2/detection_loss‚2.655282391442193 0loss/FRt3/assignment_loss‚3.3398035656322134 _step‚320 0loss/FBt1/assignment_loss‚0.2776524288856308 /loss/FRt3/detection_loss‚2.3467824079773645 .loss/FBt4/detection_loss‚0.599880376242153 _runtime‚3088.341609919 /loss/FRt2/assignment_loss‚4.430783596038818 0loss/FRt4/assignment_loss‚1.9580395072698593 /loss/FRt4/detection_loss‚1.7938765990906036 0loss/SRqqt2/detection_loss‚2.549178785747952 2loss/SRqqt1/assignment_loss‚1.6907695224508643À¡Æ>\‘Ò‚(:† †¼äÔØ®þ} gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜóÚtÓ‚(:é †¼äÔô¬¥€ disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2693.1953125 cpu‚25.342820257307775 disk.sda2.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 247.2109375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125& disk.nvme0c0n1.out‚13355.439453125 disk.drbd2265.in‚0 network.recv‚ 1.18702e+06) disk./.usagePercent‚17.19460676915051! disk.dm-0.out‚13355.439453125 disk.sda.out‚ 247.2109375$ memory_percent‚96.11482620239258 proc.cpu.threads‚17 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0$ disk.nvme0n1.out‚13355.439453125 disk.sda2.out‚0 disk.sda3.out‚ 247.2109375* proc.memory.percent‚16.437959671020508 network.sent‚ 8.424297e+06 disk.rbd1.in‚0$ disk./.usageGB‚256.2120666503906 disk.rbd1.out‚ 1.533203125 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0& proc.memory.availableMB‚ 636.546875Bjt&Ô‚(:’ •¼äÔúá¹~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.409' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.681800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T³OÞuÕ‚(:ê •¼äÔÿé±€ network.sent‚ 8.532773e+06 disk.sda.in‚ 126.77734375 disk.sda.out‚ 247.71875 disk.sda1.in‚0& disk.nvme0n1.out‚13418.90869140625$ memory_percent‚96.11806869506836* disk./.usagePercent‚17.198700073615477 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 247.71875 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2693.1953125 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 network.recv‚ 1.193123e+06 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13418.90869140625 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375# disk.dm-0.out‚13418.90869140625 proc.cpu.threads‚17$ disk./.usageGB‚256.2730598449707 disk.rbd0.in‚0 disk.sda3.out‚ 247.71875 disk.drbd2080.out‚0* proc.memory.percent‚16.437959671020508 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125& proc.memory.availableMB‚ 636.015625 cpu‚25.341495701252482Ó’Ö‚(:‡ ¤¼äÔ¶Œ¡~ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 <Mvׂ(:ë ¤¼äÔáªá€ disk.dm-1.out‚ 248.53125 disk.sda.out‚ 248.53125 disk.sda1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚13469.23681640625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2693.1953125( disk.nvme0c0n1.out‚13469.23681640625 disk.drbd2266.out‚0* disk./.usagePercent‚17.199907405146956 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125$ disk./.usageGB‚256.2910499572754 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0$ memory_percent‚96.11501693725586 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 248.53125& proc.memory.availableMB‚ 636.515625 proc.cpu.threads‚17* proc.memory.percent‚16.437959671020508 cpu‚25.340020129091457 network.sent‚ 8.538192e+06# disk.dm-0.out‚13469.23681640625 disk.rbd1.out‚ 1.533203125 disk.sda2.in‚0 network.recv‚ 1.197298e+06 disk.drbd2080.in‚0i&2Ø- _wandbruntime‚3124  _runtime‚3124 º~$ÈÙ ' lr-AdamW/pg1‚0.00014364197529228385 ' lr-AdamW/pg2‚0.00014364197529228385 trainer/global_step‚3949 % _timestamp‚1.7884196250442083e+09 _runtime‚ 3124.78175538 _step‚321¢éêËÍÚÇ ' lr-AdamW/pg1‚0.00014364197529228385 ' lr-AdamW/pg2‚0.00014364197529228385 trainer/global_step‚3949 % _timestamp‚1.7884196250442083e+09 _runtime‚ 3124.78175538 _step‚321Áx¿H2Û- _wandbruntime‚3124  _runtime‚3124~S“ÚÝ'Ü×O ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52436946 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131951541 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _runtime‚3124.782652153 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250699558776320 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3294513221826 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74538411 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚322 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3306687654400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254169653091840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884196250449207e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232878961030144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3294513221826 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18937961 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74538411 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3306687654400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232879635395584 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226852704342590 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _step‚322 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3306687654400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33498558 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254169834364416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚3949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254169653091840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884196250449207e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3470251648000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57413130 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131971581 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250699582716416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33498556 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52436946 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131951541 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3295167659582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229572273375744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232879635395584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232878961030144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3470094315520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _runtime‚3124.782652153 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229572273375744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57413132 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚2506995587763ŒªË20ÂQêt…2Þ- _wandbruntime‚3125  _runtime‚3125hÜ`]ç&ßáM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254232162077184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884196257327998e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52449679 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226907753430998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232936133496832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3125.471018777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250761209513984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229628626414592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚3949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33506726 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944/#½Vì&àæM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229628644417536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74576929 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3470952563200 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57427146 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884196257327998e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229628626414592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52449679 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226907753430998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57427148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74556865 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18942508 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232936133496832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚254232349627904 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131984011 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3295984094754 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3308163458048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚3125.471018777 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232936807875584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250761209513984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229628626414592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3307507082240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚223612423774208 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3295329656790 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚3949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18942953 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33506726 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132004077 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74556865 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚254232162077184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132004077 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52449232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884196619350195e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚ 3161.67281234 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133578393 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235738273143808 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58113930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58113930 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229638746209998 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚257295239930880 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253782119599616 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226302726171648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3336002998990 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75464463 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚229638074732882 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232389943321088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_»EHõQtrain_batch_start/allocated_bytes.large_pool.freed‚232389925318144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75464463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232389943321088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232389925318144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235737598778368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75464463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53070874 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226302743211008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232389943321088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133598433 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7884196619350195e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19164371 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58113932 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3513120331264 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚ 3161.67281234Æzpþ2î- _wandbruntime‚3162  _runtime‚3162Üñ„æ&ïàM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚3162.360442589 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75482917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232446278356992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235795445623808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133610863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75482917 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19168466 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226357630302208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232446296359936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257357741453312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229694466775730 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788419662622412e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58127948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19168922 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ÿ (+“!ðåM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3349149263872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚253843770337280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253843746397184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133610863 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75502981 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚232446296359936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235794771245056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235795445623808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257357741453312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚229694466775730 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788419662622412e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚229693795298406 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33915124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133630929 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58127948 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58127948 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19168922 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚3162.360442589 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75482917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58127946 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚232446278356992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚232446278356992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226357647341568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3336819434162 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133630929 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232446296359936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3348492888064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53083590 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257357551805440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235795445623808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75502981 D6DeviceStatsMonitor.on_train_batch_end/acti\ ±Xve.all.freed‚ 133610863 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75482917 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19168466 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3513971116032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226357630302208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235207595263488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232424116600290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389315628544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19390799 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19391246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235207577260544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229047949739008 trainer/global_step‚4049 _step‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58814730 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238596910892032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376838338398 % _timestamp‚1.7884196989605744e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58814732 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229047932699648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238596910892032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238596910892032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3388659266048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260420732319744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376838338398 % _timestamp‚1.7884196989605744e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58814732 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229047932699648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚238596910892032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34315358 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238596236526592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚235207595263488 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚235207595263488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238596236526592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232424116600290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3389315628544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19390799 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19391246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚235207577260544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3389315628544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3376183900642 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844ʯ[|2€- _wandbruntime‚3199  _runtime‚3199Jkœð àM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53719364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377654773570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4049 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260483238763520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated¡hÈqÈ_bytes.large_pool.freed‚235263930299392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58828748 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34323524 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 3199.38542553 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746MFñ¤ë&‚åM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚256926283280384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚235263930299392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58828748 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135237715 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3556761640960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232479837165814 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19395387 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229102836830208 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚331 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58828748 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260483044921344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34323524 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135257781 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚235263948302336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135257781 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3556931543040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232480508643138 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34323526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19395838 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚235263930299392 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76429033 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚ 3199.38542553 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3390135069696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3389478693888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58828746 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3390135069696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3389478693888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58828746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53719364 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377654773570 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4049 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238653408993280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260483238763520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135237715 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53718911 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76429033 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229102853869568 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚238654083372032 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4099 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3598898684416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235210158467698 _step‚334 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34723756 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59515530 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263545906583552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 % _timestamp‚1.7884197356612835e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54340754 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235210829944814 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77316567 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59515530 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚3235.399022898 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚241455548640256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259947193366016 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59515532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77316567 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238025229202944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59515532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77336605 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77316567 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238025247205888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34723758 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚241454874274816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4099 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238025229202944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3598898684416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235210158467698 _step‚334 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231793139227648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚241455548640256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34723756 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚241454874274816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3430301434368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59515530 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263545906583552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3598737157632 % _timestamp‚1.7884197356612835e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77336605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3430301434368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3417673677806 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136832097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54340754 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235210829944814 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3429645071872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77316567 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÎúC#2- _wandbruntime‚3236  _runtime‚3236K,ËØç&‘áM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263608596146176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241512721120256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚335 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77335021 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884197363432536e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3236.081352827 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59529546 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77335021 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3599752042496 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241512046741504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4099 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238081582241792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19622025 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136864567 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241512046741504 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260008820163584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235266550510546 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34731924 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 202230784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚4099 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54353508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34731926 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238081582241792 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚238081600244736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77355085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136864567 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136884633 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59529548 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚241512721120256 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59529546 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚335 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3599573751808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235265879033222 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77335021 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260008844103680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54353951 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238081582241792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884197363432536e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238081600244736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178290688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231848043358208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136884633 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3236.081352827 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59529548 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19621584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59529546 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ÏŽ4ºB2“- _wandbruntime‚3236  _runtime‚3236ÿŒ–“W”Ñ 0loss/FRt1/assignment_loss‚3.6272695452667945 /loss/SRqqt1/detection_loss‚1.77440751085476 trainer/global_step‚4099 % _timestamp‚1.7884197363450205e+09 2loss/SRqqt4/assignment_loss‚0.9400825913481671 .loss/FRt1/detection_loss‚2.167158986246863 /loss/FRt4/detection_loss‚1.8900836396217346 1loss/SRqqt4/detection_loss‚1.4035824214204633 .loss/FBt2/detection_loss‚2.211957520246506 /loss/FRt2/assignment_loss‚4.378289414174629 2loss/SRqqt1/assignment_loss‚1.6904338267536796 2loss/SRqqt2/assignment_loss‚1.9134395899517196 /loss/FRt3/detection_loss‚2.0197723808347203 .loss/FBt1/detection_loss‚2.598842058862959 _runtime‚3236.082462542 /loss/FRt3/assignment_loss‚3.032027265087503 0loss/FRt4/assignment_loss‚1.7788213312625885 /loss/FRt2/detection_loss‚2.8067995978124216 %loss/total_loss‚45.71021553281214 epoch‚3 0loss/FBt3/assignment_loss‚0.2554733650561805 0loss/SRqqt2/detection_loss‚2.667955911585263 /loss/FBt4/detection_loss‚0.8228704548657757 0loss/FBt4/assignment_loss‚0.1598476197256379 1loss/SRqqt3/detection_loss‚1.5685154385211175 2loss/SRqqt3/assignment_loss‚1.7242461861388987 0loss/FBt2/assignment_loss‚0.4972154824063125 /loss/FBt3/detection_loss‚3.1882983843485513 0loss/FBt1/assignment_loss‚0.5928250104188919 _step‚336 ]m¯p•Ö 0loss/SRqqt2/detection_loss‚2.667955911585263 /loss/FBt4/detection_loss‚0.8228704548657757 0losÅñàøìs/FBt4/assignment_loss‚0.1598476197256379 1loss/SRqqt3/detection_loss‚1.5685154385211175 2loss/SRqqt3/assignment_loss‚1.7242461861388987 0loss/FBt2/assignment_loss‚0.4972154824063125 /loss/FBt3/detection_loss‚3.1882983843485513 0loss/FBt1/assignment_loss‚0.5928250104188919 _step‚336 0loss/FRt1/assignment_loss‚3.6272695452667945 /loss/SRqqt1/detection_loss‚1.77440751085476 trainer/global_step‚4099 % _timestamp‚1.7884197363450205e+09 2loss/SRqqt4/assignment_loss‚0.9400825913481671 .loss/FRt1/detection_loss‚2.167158986246863 /loss/FRt4/detection_loss‚1.8900836396217346 1loss/SRqqt4/detection_loss‚1.4035824214204633 .loss/FBt2/detection_loss‚2.211957520246506 /loss/FRt2/assignment_loss‚4.378289414174629 2loss/SRqqt1/assignment_loss‚1.6904338267536796 2loss/SRqqt2/assignment_loss‚1.9134395899517196 /loss/FRt3/detection_loss‚2.0197723808347203 .loss/FBt1/detection_loss‚2.598842058862959 _runtime‚3236.082462542 /loss/FRt3/assignment_loss‚3.032027265087503 0loss/FRt4/assignment_loss‚1.7788213312625885 /loss/FRt2/detection_loss‚2.8067995978124216 %loss/total_loss‚45.71021553281214 epoch‚3 0loss/FBt3/assignment_loss‚0.2554733650561805ÐGät’–‚(:‡ œ½äÔä§ÿ} gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;$Óâu—‚(:ê œ½äÔ¦À¦€ disk.sda1.in‚0$ memory_percent‚96.11630439758301" proc.memory.rssMB‚ 2693.1953125 disk.sda3.out‚ 253.69140625 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-0.out‚13917.51171875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.33203125% disk.nvme0c0n1.out‚13917.51171875 disk.drbd2266.out‚0) disk./.usagePercent‚17.20755230213444 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 1.681640625' proc.memory.availableMB‚ 636.3046875* proc.memory.percent‚16.437959671020508 cpu‚25.329295584996533 network.recv‚ 1.236931e+06 disk.drbd2080.in‚0 disk.dm-1.out‚ 253.69140625 proc.cpu.threads‚17 disk.sda.out‚ 253.69140625 network.sent‚ 8.89336e+06# disk.nvme0n1.out‚13917.51171875 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125$ disk./.usageGB‚256.4049644470215 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375£á$’˜‚(:‡ «½äÔ“ž¦~ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.791' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì½ w~™‚(:ó «½äÔÉ”µ€$ disk./.usageGB‚256.4122848510742 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 254.55078125 disk.drbd2080.in‚0 disk.dm-1.out‚ 254.55078125* disk./.usagePercent‚17.208043580592648 disk.drbd2265.in‚0" disk.dm-0.out‚13976.7529296875 disk.sda.in‚ 126.77734375 network.sent‚ 9.001791e+06' disk.nvme0c0n1.out‚13976.7529296875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375$ memory_percent‚96.11630439758301" proc.memory.rssMB‚ 2693.1953125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.7138671875 disk.sda.out‚ 254.55078125' proc.memory.availableMB‚ 636.3046875 disk.sda1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚13976.7529296875 disk.drbd2265.out‚0* proc.memory.percent‚16.437959671020508 disk.drbd1716.in‚0 cpu‚25.328016854350206 network.recv‚ 1.242657e+06 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 4.33203125 proc.cpu.threads‚17¢{®’š‚(:‡ º½äÔšš“~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.724' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J½›‚(:ô º½äÔ•Ë©€% disk./.usageGB‚256.42687606811523 disk.sda3.in‚ 126.77734375 disk.sda2.out‚0( proc.memory.availableMB‚ 636.52734375" proc.memory.rssMB‚ 2693.1953125 proc.cpu.threads‚17 disk.sda3.out‚ 255.04296875 disk.nvme0c0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚14016.5693359375 network.recv‚ 1.24723e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 network.sent‚ 9.007923e+06 disk.drbd2265.in‚0 disk.dm-1.out‚ 255.04296875 disk.rbd1.in‚0 disk.sda.out‚ 255.04296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0" disk.dm-0.out‚14016.5693359375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125* proc.memory.percent‚16.437959671020508 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 1.7138671875 cpu‚25.326753160609552* disk./.usagePercent‚17.209022809411113 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.11494541168213 disk.dm-0.in‚ 4.33203125% disk.nvme0n1.out‚14016.5693359375”v"Æ2œ- _wandbruntime‚3272  _runtime‚3272%~È $ _timestamp‚1.788419772694414e+09 _runtime‚3272.431971173 _step‚337 ' lr-AdamW/pg1‚0.00014008161232262985 ' lr-AdamW/pg2‚0.00014008161232262985 trainer/global_step‚4149•»ºtÍžÇ ' lr-AdamW/pg2‚0.00014008161232262985 trainer/global_step‚4149 $ _timestamp‚1.788419772694414e+09 _runtime‚3272.431971173 _step‚337 ' lr-AdamW/pg1‚0.00014008161232262985Ñ໌2Ÿ- _wandbruntime‚3272  _runtime‚3272&÷kÀÝ' ×O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884197726951401e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244314186388480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240842899148288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234538362795008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78262657 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138458949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78242619 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚4149 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263029706309120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138478989 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3272.432900157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240842881145344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414ß⬌â'¡ÜO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244313512023040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78242619 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263029706309120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237996871812222 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244314186388480 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138478989 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237996200335106 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚234538345755648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3272.432900157 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240842881145344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240842881145344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266671518796800 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240842899148288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3471287240192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60216330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚266671335427072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884197726951401e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60216332 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244314186388480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3641629117952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240842899148288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234538362795008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3457854579458 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78262657 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3470630877696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78262657 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244313512023040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54974969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138458949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60216330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35132156 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78242619 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35132158 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _step‚338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3641788547584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚263029730249216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚4149Òtœ&’2¢- _wandbruntime‚3273  _runtime‚3273E–왣áM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234593249886208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244371358868480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚339 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60230346 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 % _timestamp‚1.7884197733956475e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60230346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on•1áN_train_batch_end/allocation.all.freed‚ 138491419 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78281137 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238051920900630 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240899234184192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266734042327040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚3273.133628371 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78281137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19847360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240899234184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944‘ƒ²®ì&¤æM P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60230348 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240899234184192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60230348 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54987686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244370684489728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19846944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚238052592377954 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234593249886208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244371358868480 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚240899252187136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138491419 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚263091357046784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 146833408 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚339 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60230346 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78261073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35140326 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234593266925568 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚263091380986880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3471450305536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3472106681344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3459325452386 % _timestamp‚1.7884197733956475e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138511485 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3471450305536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60230346 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244371358868480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78261073 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138491419 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78281137 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240899252187136 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚238051920900630 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240899234184192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138511485 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54987268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266734042327040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458671014422 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 _runtime‚3273.133628371 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78281137 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19847360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266733871553536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ó]Lˆ2¥- _wandbruntime‚3273  _runtime‚3273É_žÈZ¦Ô 1loss/SRqqt1/detection_loss‚1.7798542242783766 1loss/SRqqt4/detection_loss‚1.1614379135563093 0loss/FBt1/assignment_loss‚0.5387850222821596 0loss/SRqqt3/detection_loss‚1.893891070847926 0loss/FRt4/assignment_loss‚1.5069669385751088 0loss/SRqqt4/assignment_loss‚1.03906262531767 epoch‚3 0loss/FRt2/assignment_loss‚3.9767927885055543 /loss/FRt1/detection_loss‚2.5620508137586953 .loss/FBt1/detection_loss‚2.512420211519514 /loss/FBt3/detection_loss‚3.4720352540413537 trainer/global_step‚4149 $ _timestamp‚1.788419773397575e+09 0loss/FRt3/assignment_loss‚3.1063634134688467 2loss/SRqqt1/assignment_loss‚1.3541222161207445 -loss/FRt3/detection_loss‚2.17936484093936 1loss/SRqqt2/detection_loss‚2.7993031293153763 /loss/FBt2/detection_loss‚2.6216126588674693 _runtime‚3273.135023855 0loss/FBt4/assignment_loss‚0.2609554329079754 /loss/FBt4/detection_loss‚0.6954171053451917 &loss/total_loss‚47.340175937615875 /loss/FRt4/detection_loss‚1.9868780199127893 _step‚340 0loss/FRt1/assignment_loss‚3.6831064320899345 2loss/SRqqt3/assignment_loss‚1.2201150574761888 0loss/FBt2/assignment_loss‚0.4678185364372124 2loss/SRqqt2/assignment_loss‚2.4917145346601806 1loss/FBt3/assignment_loss‚0.47538260921282927 /loss/FRt2/detection_loss‚3.5547250881791115þá†Ú_§Ù 1loss/SRqqt4/detection_loss‚1.1614379135563093 0loss/FBt1/assignment_loss‚0.5387850222821596 0loss/SRqqt3/detection_loss‚1.893891070847926 0loss/FRt4/assignment_loss‚1.5069669385751088 0loss/SRqqt4/assignment_loss‚1.03906262531767 epoch‚3 0loss/FRt2/assignment_loss‚3.9767927885055543 /loss/FRt1/detection_loss‚2.5620508137586953 .loss/FBt1/detection_loss‚2.512420211519514 /loss/FBt3/detection_loss‚3.4720352540413537 trainer/global_step‚4149 $ _timestamp‚1.788419773397575e+09 0loss/FRt3/assignment_loss‚3.1063634134688467 2loss/SRqqt1/assignment_loss‚1.3541222161207445 -loss/FRt3/detection_loss‚2.17936484093936 1loss/SRqqt2/detection_loss‚2.7993031293153763 /loss/FBt2/detection_loss‚2.6216126588674693 _runtime‚3273.135023855 0loss/FBt4/assignment_loss‚0.2609554329079754 /loss/FBt4/detection_loss‚0.6954171053451917 &loss/total_loss‚47.340175937615875 /loss/FRt4/detection_loss‚1.9868780199127893 _step‚340 0loss/FRt1/assignment_loss‚3.6831064320899345 2loss/SRqqt3/assignment_loss‚1.2201150574761888 0loss/FBt2/assignment_loss‚0.4678185364372124 2loss/SRqqt2/assignment_loss‚2.4917145346601806 1loss/FBt3/assignment_loss‚0.47538260921282927 /loss/FRt2/detection_loss‚3.5547250881791115 1loss/SRqqt1/detection_loss‚1.7798542242783766Ô.>jô’¨‚(:‡ ɽäÔÝ‹¸~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1084 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ówt©‚(:é ɽäÔçÛð€ disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0 disk.drbd1716.in‚0 network.recv‚ 1.251139e+06# disk./.usageGB‚256.434024810791 disk.nvme0n1.in‚ 4.33203125 disk.drbd1716.out‚0* proc.memory.percent‚16.437959671020508 cpu‚25.325490494564853 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚14084.1142578125 disk.drbd2080.out‚0 disk.sda1.out‚0$ memory_percent‚96.11806869506836 network.sent‚ 9.013759e+06 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 255.84375 disk.sda.in‚ 126.77734375* disk./.usagePercent‚17.209502567530283 disk.sda2.out‚0 disk.drbd2265.out‚0" disk.dm-0.out‚14084.1142578125 disk.drbd2266.in‚0 disk.sda.out‚ 255.84375 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 1.77685546875 disk.sda1.in‚0& proc.memory.availableMB‚ 636.015625 disk.drbd2265.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2693.1953125 proc.cpu.threads‚17 disk.sda3.out‚ 255.84375% disk.nvme0n1.out‚14084.1142578125 disk.rbd0.out‚0tªY ’ª‚(:‡ ؽäÔ¹ ˜~ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02_· }«‚(:ò ؽäÔêò– disk.drbd2080.out‚0 disk.dm-1.out‚ 256.33984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0* proc.memory.percent‚16.437959671020508 disk.rbd1.out‚ 1.77685546875 disk.sda1.out‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚14123.4619140625 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 4.33203125 proc.cpu.threads‚17 network.recv‚ 1.25788e+06 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2693.1953125 disk.drbd2265.in‚0" disk.dm-0.out‚14123.4619140625 disk.rbd1.in‚0 disk.sda.out‚ 256.33984375 network.sent‚ 9.122719e+06* disk./.usagePercent‚17.210673289984364$ disk./.usageGB‚256.4514694213867 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0% disk.nvme0n1.out‚14123.4619140625 disk.rbd0.out‚0 disk.sda3.out‚ 256.33984375$ memory_percent‚96.11959457397461& proc.memory.availableMB‚ 635.765625 cpu‚25.324168692883877ÊH!2¬- _wandbruntime‚3309  _runtime‚3309ÐóÅAÉ­Ã ' lr-AdamW/pg1‚0.00013916201215856863 ' lr-AdamW/pg2‚0.00013916201215856863 trainer/global_step‚4199 % _timestamp‚1.7884198097603023e+09 _runtime‚3309.497867764 _step‚341•œõÎ®È ' lr-AdamW/pg2‚0.00013916201215856863 trainer/global_step‚4199 % _timestamp‚1.7884198097603023e+09 _runtime‚3309.497867764 _step‚341 ' lr-AdamW/pg1‚0.00013916201215856863Õ¾?æk2¯- _wandbruntime‚3309  _runtime‚3309] ?Ü'°ÖO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180401152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60917132 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 204341248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚243660533087744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266112267132416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512273046016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚458 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60917130 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79168671 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚243660551090688 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚243660533087744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511616683520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20068567 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79188709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3511616683520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884198097610385e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55609125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3499344356622 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚ 3309.49880871 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3511616683520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884198097610385e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512273046016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QDDeviceStatsMonitor.on_train_batch_start/inactive_split«ùü[ .all.allocated‚55609125 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3499344356622 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚247172149771264 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚247172824136704 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79168671 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3684566437376 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3498689918866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚342 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35540556 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237283552283648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20068109 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269796833569792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚ 3309.49880871 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247172149771264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269796629228544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140085801 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55608665 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3684386036224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240782913679630 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79188709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60917130 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884198104486113e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247229996616704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140118271 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55621756 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚343 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269859145346048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247229322237952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60931146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512436111360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3499506353830 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247229996616704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79207189 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240838634245362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79187125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512436111360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1ã(Fì&´æM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚247229996616704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79207189 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚240838634245362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79187125 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512436111360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79207189 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35548724 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79187125 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140118271 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚243716886126592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884198104486113e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚247229996616704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140118271 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55621756 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60931148 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140138337 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513092487168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140138337 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20072601 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55621325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚343 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269859145346048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266173893929984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500160791794 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚247229322237952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60931146 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237338456414208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚4199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3512436111360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3499506353830 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928×51Þ2µ- _wandbruntime‚3310  _runtime‚3310œYí^¶Ø /loss/FRt2/assignment_loss‚3.713579288235417 2loss/SRqqt2/assignment_loss‚2.2619525376827485 /loss/FBt1/detection_loss‚2.3189593060658527 0loss/FRt1/assignment_loss‚3.5995867432549944 1loss/SRqqt2/detection_loss‚2.6509894853638065 1loss/SRqqt3/detection_loss‚1.7619137563267533 .loss/FRt2/detection_loss‚3.225012355380588 epoch‚3 trainer/global_step‚4199 2loss/FBt3/assignment_loss‚0.033582364581525324 /loss/FRt4/detection_loss‚1.6882060442918114 1loss/SRqqt4/detection_loss‚1.1698958687484264 .loss/FBt2/detection_loss‚3.549921289086342 /loss/FBt4/detection_loss‚0.8599830749812291 _step‚344 0loss/FBt1/assignment_loss‚0.8405319211574696 1loss/FBt2/assignment_loss‚0.16675190185195987 /loss/FRt3/assignment_loss‚3.360786333680153 2loss/SRqqt4/assignment_loss‚1.0101276162400175 /loss/FRt1/detection_loss‚2.4668814886447996 /loss/FBt3/detection_loss‚2.7943128824234007 % _timestamp‚1.7884198104504282e+09 /loss/FRt4/assignment_loss‚1.746929453951972 2loss/SRqqt3/assignment_loss‚1.3067247060184577 1loss/FBt4/assignment_loss‚0.23392043501256074 .loss/FRt3/detection_loss‚2.195309305563569 1loss/SRqqt1/detection_loss‚2.0618506295736445 _runtime‚3310.187887547 2loss/SRqqt1/assignment_loss‚1.3678419182813444 &loss/total_loss‚46.385550706398845qô:c·Ý 0loss/FBt1/assignment_loss‚0.8405319211574696 1loss/FBt2/assignment_loss‚0.16675190185195987 /loss/FRt3/assignment_loss‚3.360786333680153 2loss/SRqqt4/assignment_loss‚1.0101276162400175 /loss/FRt1/detection_loss‚2.4668814886447996 /loss/FBt3/detection_loss‚2.7943128824234007 % _timestamp‚1.7884198104504282e+09 /loss/FRt4/assignment_loss‚1.746929453951972 2loss/SRqqt3/assignment_loss‚1.3067247060184577 1loss/FBt4/assignment_loss‚0.23392043501256074 .loss/FRt3/detection_loss‚2.195309305563569 1loss/SRqqt1/detection_loss‚2.0618506295736445 _runtime‚3310.187887547 2loss/SRqqt1/assignment_loss‚1.3678419182813444 &loss/total_loss‚46.385550706398845 /loss/FRt2/assignment_loss‚3.713579288235417 2loss/SRqqt2/assignment_loss‚2.2619525376827485 /loss/FBt1/detection_loss‚2.3189593060658527 0loss/FRt1/assignment_loss‚3.5995867432549944 1loss/SRqqt2/detection_loss‚2.6509894853638065 1loss/SRqqt3/detection_loss‚1.7619137563267533 .loss/FRt2/detection_loss‚3.225012355380588 epoch‚3 trainer/global_step‚4199 2loss/FBt3/assignment_loss‚0.033582364581525324 /loss/FRt4/detection_loss‚1.6882060442918114 1loss/SRqqt4/detection_loss‚1.1698958687484264 .loss/FBt2/detection_loss‚3.549921289086342 /loss/FBt4/detection_loss‚0.8599830749812291 _step‚344Ø'‰€’¸‚(:‡ ç½äÔÝŽ¬~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.828' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5656 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥ô1T¹‚(:ô ç½äÔí¾Ñ€ network.recv‚ 1.261207e+06( disk.nvme0c0n1.out‚14182.61962890625 disk.dm-1.out‚ 257.11328125 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚14182.61962890625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2693.1953125* proc.memory.percent‚16.437959671020508% disk./.usageGB‚256.46500396728516 disk.drbd2080.out‚0' proc.memory.availableMB‚ 636.4765625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚96.11525535583496 cpu‚25.32294609206049 disk.sda.out‚ 257.11328125 network.sent‚ 9.127992e+06 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125* disk./.usagePercent‚17.211581604715846 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 257.11328125 disk.drbd2265.out‚0# disk.dm-0.out‚14182.61962890625 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.83984375j£©Vº‚(:’ ö½äÔˆª~ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocaWçGGted‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.732' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.946399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\bà•~»‚(:ó ö½äÔ¬ùÆ disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.out‚ 257.88671875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd1.out‚ 1.83984375& disk.nvme0n1.out‚14225.27978515625 disk.rbd0.in‚0 disk.sda3.out‚ 257.88671875" proc.memory.rssMB‚ 2693.1953125 cpu‚25.32171184005485 proc.cpu.threads‚17 network.recv‚ 1.267811e+06 disk.dm-1.out‚ 257.88671875 disk.drbd2266.in‚0 disk.sda1.out‚0% disk./.usageGB‚256.47057723999023 disk.drbd2080.in‚0 disk.drbd2266.out‚0$ memory_percent‚96.11678123474121 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚17.21195563172337 disk.drbd2265.in‚0# disk.dm-0.out‚14225.27978515625' proc.memory.availableMB‚ 636.2265625 network.sent‚ 9.236622e+06 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚14225.27978515625* proc.memory.percent‚16.437959671020508äb€’¼‚(:‡ …¾äԧ݇~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.297' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È£Õ™}½‚(:ò …¾äÔ‹¦ € disk.sda3.out‚ 258.46484375* disk./.usagePercent‚17.212329402723363 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0" proc.memory.rssMB‚ 2693.1953125 network.sent‚ 9.242563e+06 network.recv‚ 1.271942e+06 disk.nvme0c0n1.in‚ 4.33203125 disk.sda.out‚ 258.46484375 disk.drbd1716.out‚0% disk./.usageGB‚256.47614669799805& disk.nvme0n1.out‚14283.34814453125 disk.sda2.in‚0 disk.dm-1.out‚ 258.46484375 disk.drbd2266.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd1716.in‚0$ proc.memory.availableMB‚636.1875# disk.dm-0.out‚14283.34814453125 disk.rbd1.in‚0* proc.memory.percent‚16.437959671020508 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚14283.34814453125 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375$ memory_percent‚96.11701965332031 cpu‚25.320374736418486 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0ß`´æ2¾- _wandbruntime‚3346  _runtime‚3346Ÿõ«¬È¿Â ' lr-AdamW/pg1‚0.00013823098785776432 ' lr-AdamW/pg2‚0.00013823098785776432 trainer/global_step‚4249 $ _timestamp‚1.788419846278398e+09 _runtime‚3346.016022151 _step‚345xD˜TÍÀÇ ' lr-AdamW/pg1‚0.00013823098785776432 ' lr-AdamW/pg2‚0.00013823098785776432 trainer/global_step‚4249 $ _timestamp‚1.788419846278398e+09 _runtime‚3346.016022151 _step‚345ÙËÕä 2Á- _wandbruntime‚3346  _runtime‚3346¯ ¬ñÝ'Â×O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56243176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269194804015616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 trainer/global_step‚4249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250031461884928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884198462791417e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80114761 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3553258851840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144¯Ÿx:â'ÃÜO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272922222898176 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250031461884928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250030787519488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80114761 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3727251064320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚346 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61617930 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3553258851840 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246478185030144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243568284069922 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80114761 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56243176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272922031139840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61617930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚269194804015616 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246478203033088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61617932 trainer/global_step‚4249 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3539525258274 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56243626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35948958 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240028758811648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250031461884928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20294220 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80094723 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727418882560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240028775851008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884198462791417e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35948956 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141732693 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚269194780075520 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141745123 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61631948 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3554078292992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61631946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3346.730413437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56256157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269256430813184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3728101649408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4249 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884198469923613e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240083662942208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚347 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936ÞGôì&ÆæM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250088634364928 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141765189 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540341693238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61631948 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61631946 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3554078292992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3554078292992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61631946 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56255713 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3346.730413437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35957126 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56256157 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚269256430813184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3728101649408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141765189 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚4249 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80133241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884198469923613e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80113177 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240083662942208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚347 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246534556071936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250087959986176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272984532462592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141745123 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80113177 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246534538068992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20299031 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141745123 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61631948 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240083679981568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890099633152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246354325937330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36357358 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62318732 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272277316958720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3770229218304 _runtime‚3383.144946143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884198834071152e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928øHì6àÓÜO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3593588295168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143359545 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3593588295168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252889425267712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272277340898816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3580360597682 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249295836972544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242773965339648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276047546177024 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56877310 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252889425267712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3770397036544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276047737935360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20519501 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252890099633152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62318730 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚246354325937330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚249295836972544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36357358 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242773982379008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36357356 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62318730 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_al<¹ê^locations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62318732 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249295854975488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚252890099633152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3594244657664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272277316958720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3770229218304 _runtime‚3383.144946143 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143339505 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81040813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚246354997414446 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7884198834071152e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62318732 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56876857 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20519952 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81020775 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249295854975488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272338967696384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249352208014336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚351 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62332746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276110047450624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 3383.85017981 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20524489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62332748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884198841120021e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242828869470208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246410717980178 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¿LJë&ÖåM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56889568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276110047450624 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81059293 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3581831470610 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚4299 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143371975 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚ 3383.85017981 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62332746 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3595064098816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20524489 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3771249656320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249352190011392 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62332748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81059293 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272338991636480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚246410046502854 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62332748 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252947272113152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884198841120021e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242828869470208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276110241292800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3594407723008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚249352190011392 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚246410717980178 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143392041 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272338967696384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚249352208014336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143371975 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81039229 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚351 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚252946597734400 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143392041 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62332746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36365526 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36365524 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3581177032646 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249352208014336 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3595064098816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3771079754240 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430ß7í2×- _wandbruntime‚3383  _runtime‚3383vüx°YØÓ /loss/FBt3/detection_loss‚2.6370699206988015 %loss/total_loss‚45.07995187605992 2loss/SRqqt3/assignment_loss‚1.6617340207870663 2loss/SRqqt4/assignment_loss‚1.1816990097102367 0loss/FRt1/assignment_loss‚3.0792960185896265 1loss/FBt4/assignment_loss‚0.15011911115539522 /loss/FRt1/detection_loss‚1.9641464135863564 1loss/SRqqt2/detection_loss‚2.2897262186617464 0loss/FRt2/assignment_loss‚3.8392104598192067 /loss/FRt3/assignment_loss‚3.108036620112566 trainer/global_step‚4299 $ _timestamp‚1.788419884113857e+09 0loss/FRt4/assignment_loss‚1.4260713458061218 /loss/FBt1/assignment_loss‚0.626135834134542 .loss/FRt4/detection_loss‚1.625707654881158 1loss/SRqqt4/detection_loss‚1.1115552856047686 _runtime‚3383.851307418 _step‚352 2loss/SRqqt2/assignment_loss‚2.3707936845115714 0loss/FBt2/assignment_loss‚1.2204386790564665 /loss/FBt1/detection_loss‚1.8832996407380471 2loss/SRqqt1/assignment_loss‚1.2711075484612078 .loss/FRt3/detection_loss‚2.400633076062569 1loss/SRqqt1/detection_loss‚1.7532062165017397 /loss/FBt4/detection_loss‚0.8693069834472327 0loss/SRqqt3/detection_loss‚1.805314238729148 /loss/FBt2/detection_loss‚3.1085708340009055 epoch‚3 0loss/FBt3/assignment_loss‚0.5314567933479944 .loss/FRt2/detection_loss‚3.165316267655446ÊÁ/ü^ÙØ 0loss/FRt2/assignment_loss‚3.8392104598192067 /loss/FRt3/assignment_loss‚3.108036620112566 trainer/global_step‚4299 $ _timestamp‚1.788419884113857e+09 0loss/FRt4/assignment_loss‚1.4260713458061218 /loss/FBt1/assignment_loss‚0.626135834134542 .loss/FRt4/detection_loss‚1.625707654881158 1loss/SRqqt4/detection_loss‚1.1115552856047686 _runtime‚3383.851307418 _step‚352 2loss/SRqqt2/assignment_loss‚2.3707936845115714 0loss/FBt2/assignment_loss‚1.2204386790564665 /loss/FBt1/detection_loss‚1.8832996407380471 2loss/SRqqt1/assignment_loss‚1.2711075484612078 .loss/FRt3/detection_loss‚2.400633076062569 1loss/SRqqt1/detection_loss‚1.7532062165017397 /loss/FBt4/detection_loss‚0.8693069834472327 0loss/SRqqt3/detection_loss‚1.805314238729148 /loss/FBt2/detection_loss‚3.1085708340009055 epoch‚3 0loss/FBt3/assignment_loss‚0.5314567933479944 .loss/FRt2/detection_loss‚3.165316267655446 /loss/FBt3/detection_loss‚2.6370699206988015 %loss/total_loss‚45.07995187605992 2loss/SRqqt3/assignment_loss‚1.6617340207870663 2loss/SRqqt4/assignment_loss‚1.1816990097102367 0loss/FRt1/assignment_loss‚3.0792960185896265 1loss/FBt4/assignment_loss‚0.15011911115539522 /loss/FRt1/detection_loss‚1.9641464135863564 1loss/SRqqt2/detection_loss‚2.2897262186617464à:S/¿Ú‚(:… ²¾äÔ‰ä¨~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.61' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.122 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xX2š~Û‚(:ó ²¾äÔåÃÇ€ disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 260.00390625( proc.memory.availableMB‚ 635.71484375 network.recv‚ 1.28605e+06 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda.out‚ 260.00390625$ disk./.usageGB‚256.5098419189453 disk.sda3.out‚ 260.00390625 disk.drbd2080.out‚0 network.sent‚ 9.36296e+06 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2266.in‚0* proc.memory.percent‚16.437959671020508 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.33203125 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.214590717273307( disk.nvme0c0n1.out‚14411.44287109375 disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0# disk.dm-0.out‚14411.44287109375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 cpu‚25.316808347147152 disk.sda1.in‚0 disk.drbd2265.out‚0$ memory_percent‚96.11990451812744" proc.memory.rssMB‚ 2693.1953125 disk.drbd1716.in‚0& disk.nvme0n1.out‚14411.44287109375 disk.rbd0.in‚0 disk.rbd0.out‚0g›|Ñ’Ü‚(:‡ Á¾äԀЕ~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MŒ owÝ‚(:ì Á¾äÔªˆ disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375( proc.memory.availableMB‚ 635.96484375 proc.cpu.threads‚17 network.recv‚ 1.292327e+06 disk.dm-0.in‚ 4.33203125 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 260.57421875 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2265.out‚0% disk.nvme0n1.out‚14482.8818359375$ memory_percent‚96.11837863922119 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375" proc.memory.rssMB‚ 2693.1953125 cpu‚25.31556151615177 network.sent‚ 9.471458e+06% disk./.usagePercent‚ 17.2154071253 disk.drbd2266.out‚0* proc.memory.percent‚16.437959671020508 disk.nvme0c0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚14482.8818359375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ disk./.usageGB‚256.5220069885254 disk.sda3.out‚ 260.57421875" disk.dm-0.out‚14482.8818359375 disk.dm-1.out‚ 260.57421875 disk.sda3.in‚ 126.77734375~^Y’Þ‚(:‡ оäÔ¬ý~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M{Æß‚(:ì оäÔ¬ÓЀ) disk./.usagePercent‚17.21478528299933 disk.sda2.in‚0 cpu‚25.31441214401938 disk.drbd2266.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚14529.4638671875 disk.sda3.in‚ 126.77734375( proc.memory.availableMB‚ 635.95703125 network.sent‚ 9.477808e+06 network.recv‚ 1.29702e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda.out‚ 260.9921875 disk.sda3.out‚ 260.9921875" proc.memory.rssMB‚ 2693.1953125 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0* proc.memory.percent‚16.437959671020508 proc.cpu.threads‚17 disk.drbd2080.out‚0" disk.dm-0.out‚14529.4638671875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.in‚0% disk.nvme0n1.out‚14529.4638671875$ memory_percent‚96.11842632293701$ disk./.usageGB‚256.5127410888672 di·çì¯isk.dm-1.out‚ 260.9921875 disk.rbd1.out‚ 1.83984375 disk.drbd1716.out‚0 disk.drbd2265.out‚0öè2à- _wandbruntime‚3420  _runtime‚3420âéL€Éáà % _timestamp‚1.7884199203420517e+09 _runtime‚3420.079602312 _step‚353 ' lr-AdamW/pg1‚0.00013633544528254056 ' lr-AdamW/pg2‚0.00013633544528254056 trainer/global_step‚4349²Z¯ÎâÈ trainer/global_step‚4349 % _timestamp‚1.7884199203420517e+09 _runtime‚3420.079602312 _step‚353 ' lr-AdamW/pg1‚0.00013633544528254056 ' lr-AdamW/pg2‚0.00013633544528254056árš¢2ã- _wandbruntime‚3420  _runtime‚3420¶°ÑZÝ'ä×O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63019530 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3621195937090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252113488914944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3621850374846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3813118608896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249141039281854 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275359853841920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245519171867648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81946827 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81966865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57511664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249140367804738 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884199203427887e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 trainer/global_step‚4349 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255748063015936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252113506917888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872HÊâ'åÜO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144986397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57511219 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36765758 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275359853841920 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245519171867648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252113506917888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81946827 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81966865 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57511664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚249140367804738 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3634574100992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7884199203427887e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255748737381376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279173162112000 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144966357 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚279172972450816 trainer/global_step‚4349 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚255748063015936 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63019532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252113506917888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63019530 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3813284329984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3621195937090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255748063015936 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252113488914944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3634574100992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81966865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3621850374846 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63019532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3813118608896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚249141039281854 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3635230463488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275359877782016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0âî1}2æ- _wandbruntime‚3420  _runtime‚3420¤¡­©æ&çàM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249196088370262 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252169859956736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245574093037568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144998827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788419921044142e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3636049904640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 _step‚355ÕÖ).èåM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63033548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81985345 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81965281 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36773924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20749906 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3635393528832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3622012372054 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255805909861376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252169859956736 H:DeviceStatsMonitor.on_train_böã"¥l atch_end/allocation.all.freed‚ 144998827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57523830 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788419921044142e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145018893 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3636049904640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279235670302720 _step‚355 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249196759847586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255805235482624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57524276 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144998827 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63033546 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145018893 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249196088370262 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚4349 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚279235480654848 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3813976075264 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252169859956736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3636049904640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245574093037568 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3622666810018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82892917 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3662685714254 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146613249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889Ô©iiõÜO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146593209 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚258607375129600 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20972453 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3856222551552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20972894 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3856050539008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚258607375129600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3675559906816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254931140857344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚254931158860288 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258606700764160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3676216269312 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146613249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37174156 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63720332 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63720330 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3675559906816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37174158 ZLDeviceStatsMonitor.on_train_batch_starì.•y t/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251926409672146 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3662031276498 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82892917 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82872879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248264395435008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3662685714254 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82891333 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282361147175936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884199581018817e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258664547609600 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20976979 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278504041462784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254987511899136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚282360968013824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚359 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63734346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3457.840131976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚254987511899136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248319299565568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734346 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚4399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734348 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ÿšffì&øæM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚278504065402880 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3662847711462 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58159303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚278504041462784 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254987511899136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3676379334656 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚282360968013824 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚359 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63734346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82911397 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚3457.840131976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3857081773056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚254987511899136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248319299565568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63734346 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82891333 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚254987493896192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146625679 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚4399 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63734348 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146645745 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82911397 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82891333 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3677035710464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚248319282526208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254987493896192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚282361147175936 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3676379334656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3856926551040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884199581018817e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251982130237670 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚258664547609600 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20976979 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251982801714994 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605çÙK…2ù- _wandbruntime‚3457  _runtime‚3457##l\úÖ &loss/total_loss‚45.254923087775964 epoch‚3 trainer/global_step‚4399 % _timestamp‚1.7884199581040716e+09 /loss/FRt1/assignment_loss‚3.552046799659729 2loss/SRqqt3/assignment_loss‚1.4448934562247375 0loss/SRqqt3/detection_loss‚1.568965910323735 /loss/FBt1/detection_loss‚2.5572278996308646 0loss/FBt3/assignment_loss‚0.3980068499222398 0loss/SRqqt2/detection_loss‚2.373606248813517 /loss/FBt2/detection_loss‚3.2288897037506104 /loss/FRt2/assignment_loss‚4.100782949477434 /loss/FRt2/detection_loss‚2.8969837464392185 /loss/FRt4/detection_loss‚1.8239263754408315 _runtime‚3457.841496521 0loss/FRt3/assignment_loss‚2.6408991205926036 2loss/SRqqt4/assignment_loss‚1.3315076657153408 /loss/FRt1/detection_loss‚2.3344370670616628 .loss/FRt3/detection_loss‚2.175430581850164 1loss/SRqqt4/detection_loss‚1.2382330316407928 /loss/FBt3/detection_loss‚2.7046592086553574 /loss/FBt4/detection_loss‚0.6169050174842899 0loss/FRt4/assignment_loss‚1.5330424916069463 2loss/SRqqt2/assignment_loss‚1.9979295463246458 0loss/FBt1/assignment_loss‚0.7648359322019278 1loss/FBt4/assignment_loss‚0.22601065225833786 _step‚360 2loss/SRqqt1/assignment_loss‚1.5074189732350953 0loss/FBt2/assignment_loss‚0.3513460434972956 1loss/SRqqt1/detection_loss‚1.8869378159685832ý“aûÛ /loss/FRt1/detection_loss‚2.3344370670616628 .loss/FRt3/detection_loss‚2.175430581850164 1loss/SRqqt4/detection_loss‚1.2382330316407928 /loss/FBt3/detection_loss‚2.7046592086553574 /loss/FBt4/detection_loss‚0.6169050174842899 0loss/FRt4/assignment_loss‚1.5330424916069463 2loss/SRqqt2/assignment_loss‚1.9979295463246458 0loss/FBt1/assignment_loss‚0.7648359322019278 1loss/FBt4/assignment_loss‚0.22601065225833786 _step‚360 2loss/SRqqt1/assignment_loss‚1.5074189732350953 0loss/FBt2/assignment_loss‚0.3513460434972956 1loss/SRqqt1/detection_loss‚1.8869378159685832 &loss/total_loss‚45.254923087775964 epoch‚3 trainer/global_step‚4399 % _timestamp‚1.7884199581040716e+09 /loss/FRt1/assignment_loss‚3.552046799659729 2loss/SRqqt3/assignment_loss‚1.4448934562247375 0loss/SRqqt3/detection_loss‚1.568965910323735 /loss/FBt1/detection_loss‚2.5572278996308646 0loss/FBt3/assignment_loss‚0.3980068499222398 0loss/SRqqt2/detection_loss‚2.373606248813517 /loss/FBt2/detection_loss‚3.2288897037506104 /loss/FRt2/assignment_loss‚4.100782949477434 /loss/FRt2/detection_loss‚2.8969837464392185 /loss/FRt4/detection_loss‚1.8239263754408315 _runtime‚3457.841496521 0loss/FRt3/assignment_loss‚2.6408991205926036 2loss/SRqqt4/assignment_loss‚1.3315076657153408èÙl¡’ü‚(:‡ ý¾äÔ¬›ú~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xë~ý‚(:ó ý¾äÔ‰¹ª€ disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375' proc.memory.availableMB‚ 636.1015625" proc.memory.rssMB‚ 2693.1953125 proc.cpu.threads‚17 network.sent‚ 9.597485e+06 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0$ disk./.usageGB‚256.5678482055664 disk.drbd2080.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 126.77734375 disk.sda.out‚ 262.80078125 cpu‚25.31093683688864 disk.drbd1716.out‚0 disk.drbd2265.in‚0# disk.dm-0.out‚14679.14111328125 disk.sda1.in‚0 disk.sda3.out‚ 262.80078125 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚14679.14111328125 disk.sda2.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.437959671020508 network.recv‚ 1.310181e+06& disk.nvme0n1.out‚14679.14111328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 262.80078125$ memory_percent‚96.11754417419434* disk./.usagePercent‚17.218483567838966 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125E:LI’þ‚(:‡ Œ¿äÔ›¾¸~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03*÷nxÿ‚(:í Œ¿äÔš¨¹€ disk.sda1.in‚0 network.sent‚ 9.705659e+06 network.recv‚ 1.317124e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2693.1953125 cpu‚25.30981863985345 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 263.171875 disk.drbd2080.out‚0# disk.dm-0.out‚14726.03173828125 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0$ memory_percent‚96.11754417419434 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2266.in‚0' proc.memory.availableMB‚ 636.1015625 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0& disk.nvme0n1.out‚14726.03173828125( disk.nvme0c0n1.out‚14726.03173828125 disk.drbd2265.out‚0 disk.dm-1.out‚ 263.171875 disk.rbd1.out‚ 1.83984375 disk.nvme0n1.in‚ 4.33203125$ disk./.usageGB‚256.5511703491211 disk.dm-0.in‚ 4.33203125* proc.memory.percent‚16.437959671020508* disk./.usagePercent‚17.217364302899266 disk.rbd0.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 263.171875ð{+¿2€- _wandbruntime‚3494  _runtime‚3494ü(xÉà ' lr-AdamW/pg1‚0.00013439656462756162 ' lr-AdamW/pg2‚0.00013439656462756162 trainer/global_step‚4449 % _timestamp‚1.7884199942702386e+09 _runtime‚3494.007764664 _step‚361X0†DÎ‚È ' lr-AdamW/pg2‚0.00013439656462756162 trainer/global_step‚4449 % _timestamp‚1.7884199942702386e+09 _runtime‚3494.007764664 _step‚361 ' lr-AdamW/pg1‚0.00013439656462756162é„ ²õ2ƒ- _wandbruntime‚3494  _runtime‚3494×kmÅ„×O % _timestamp‚1.7884199942711222e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285424214266880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257748810802688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64421132 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83818969 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3899262718464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83798931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254712451539554 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257748792799744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_poœŒ™%Ïol.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚3494.008964198 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257748792799744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254713123016670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285424024605696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251009601963008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58782183 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3717202075136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚4449 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Hµ¸Râ'…ÜO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚261465338512384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37582558 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281524927608320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64421132 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚281524951548416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3717202075136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚4449 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58782612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21200054 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚427 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884199942711222e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285424214266880 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257748810802688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64421132 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83818969 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21199627 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148220061 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚261466012877824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3899262718464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261465338512384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83798931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254712451539554 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83818969 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3716545712640 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257748792799744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚3494.008964198 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚261466012877824 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3717202075136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚257748792799744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚254713123016670 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3716545712640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚285424024605696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251009601963008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148240101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58782183 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚363 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285486724358144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚3494.702695474 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257805163841536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3717365140480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3718021516288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148252531 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251064506093568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3704337488834 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64435146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58794844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64435146 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900122072064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64435148 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148252531 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251064506093568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251064489054208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3704337488834 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64435146 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37590726 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚363 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21204120 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚285486724358144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148272597 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83837449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148272597 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261523185357824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37590724 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚3494.702695474 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257805145838592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281586578345984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257805163841536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944ëý9”‘2‰- _wandbruntime‚3494  _runtime‚3494·|f;VŠÐ /loss/FRt2/detection_loss‚2.9088422169287997 1loss/SRqqt3/detection_loss‚1.8142954682817265 epoch‚3 /loss/FRt2/assignment_loss‚3.873210887114207 0loss/FRt4/assignment_loss‚1.5606839633455463 .loss/FRt3/detection_loss‚2.295139221584096 0loss/FBt1/assignment_loss‚0.5194578197758595 /loss/FRt4/detection_loss‚1.5370298813955456 _step‚364 0loss/FRt1/assignment_loss‚2.9743098488875797 2loss/SRqqt2/assignment_loss‚2.1020360219038348 .loss/FBt1/detection_loss‚2.445312879302285 .loss/FBt4/detection_loss‚0.742797472663889 $ _timestamp‚1.788419994966302e+09 1loss/SRqqt4/assignment_loss‚1.001376108981777 1loss/FBt2/assignment_loss‚0.48538274936597897 $loss/total_loss‚44.4822577183636 0loss/FBt3/assignment_loss‚0.4711410986201372 0loss/SRqqt2/detection_loss‚2.202671626752073 1loss/SRqqt4/detection_loss‚0.8923059042705141 /loss/FRt3/assignment_loss‚3.164917963392594 2loss/SRqqt1/assignment_loss‚1.4418402388691902 2loss/SRqqt3/assignment_loss‚1.2797374837860769 1loss/SRqqt1/detection_loss‚1.9796219068978513 /loss/FBt2/detection_loss‚3.0926027562883167 .loss/FBt3/detection_loss‚3.157604535420736 0loss/FBt4/assignment_loss‚0.2058178454816078 /loss/FRt1/detection_loss‚2.3341218190533772 trainer/global_step‚4449 _runtime‚3494.703742981³ÙÃ[‹Õ epoch‚3 /loss/FRt2/assignment_loss‚3.873210887114207 0loss/FRt4/assignment_loss‚1.5606839633455463 .loss/FRt3/detection_loss‚2.295139221584096 0loss/FBt1/assignment_loss‚0.5194578197758595 /loss/FRt4/detection_loss‚1.5370298813955456 _step‚364 0loss/FRt1/assignment_loss‚2.9743098488875797 2loss/SRqqt2/assignment_loss‚2.1020360219038348 .loss/FBt1/detection_loss‚2.445312879302285 .loss/FBt4/detection_loss‚0.742797472663889 $ _timestamp‚1.788419994966302e+09 1loss/SRqqt4/assignment_loss‚1.001376108981777 1loss/FBt2/assignment_loss‚0.48538274936597897 $loss/total_loss‚44.4822577183636 0loss/FBt3/assignment_loss‚0.4711410986201372 0loss/SRqqt2/detection_loss‚2.202671626752073 1loss/SRqqt4/detection_loss‚0.8923059042705141 /loss/FRt3/assignment_loss‚3.164917963392594 2loss/SRqqt1/assignment_loss‚1.4418402388691902 2loss/SRqqt3/assignment_loss‚1.2797374837860769 1loss/SRqqt1/detection_loss‚1.9796219068978513 /loss/FBt2/detection_loss‚3.0926027562883167 .loss/FBt3/detection_loss‚3.157604535420736 0loss/FBt4/assignment_loss‚0.2058178454816078 /loss/FRt1/detection_loss‚2.3341218190533772 trainer/global_step‚4449 _runtime‚3494.703742981 /loss/FRt2/detection_loss‚2.9088422169287997 1loss/SRqqt3/detection_loss‚1.8142954682817265ì³USVŒ‚(:’ ›¿äÔòóª~ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯®${‚(:ð ›¿äÔ´äÄ€ disk.drbd1716.out‚0) disk./.usagePercent‚17.22045866596563 disk.rbd0.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda1.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.437959671020508& disk.nvme0n1.out‚14799.23681640625 disk.sda2.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚14799.23681640625 disk.sda.out‚ 263.7421875 disk.nvme0c0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚14799.23681640625 network.recv‚ 1.320919e+06$ disk./.usageGB‚256.5972785949707 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚95.49615383148193( proc.memory.availableMB‚ 737.91015625 proc.cpu.threads‚17 disk.sda3.out‚ 263.7421875 cpu‚25.30871047397618 network.sent‚ 9.710983e+06 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 263.7421875" proc.memory.rssMB‚ 2693.1953125 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 ¶+’Ž‚(:‡ ª¿äԬʋ~ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âs‚(:è ª¿äÔšÞ†€ disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 264.34375 disk.drbd2265.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚14840.68994140625# disk.dm-0.out‚14840.68994140625 disk.dm-1.out‚ 264.34375 disk.nvme0c0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚14840.68994140625 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375" disk./.usageGB‚256.61279296875 disk.dm-1.in‚ 126.77734375' proc.memory.availableMB‚ 737.6484375" proc.memory.rssMB‚ 2693.1953125 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚95.49775123596191* proc.memory.percent‚16.437959671020508 cpu‚25.30754961376357 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 264.34375 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 network.recv‚ 1.327564e+06* disk./.usagePercent‚17.221499848607387 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.83984375 network.sent‚ 9.819854e+06T;Ö ’‚(:‡ ¹¿äÔºâ ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž.Èr|‘‚(:ñ ¹¿äÔ®¿Ù€ cpu‚25.306381680693338 disk.drbd2266.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.sent‚ 9.825793e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚14886.39208984375 disk.drbd2265.out‚0* disk./.usagePercent‚17.220017052969062 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.sda3.out‚ 265.1015625# disk.dm-0.out‚14886.39208984375 disk.dm-1.out‚ 265.1015625$ memory_percent‚95.49479484558105* proc.memory.percent‚16.437959671020508 disk.drbd2080.in‚0& disk.nvme0n1.out‚14886.39208984375 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375" proc.memory.rssMB‚ 2693.1953125 network.recv‚ 1.331473e+06$ disk./.usageGB‚256.5906982421875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 265.1015625 disk.sda3.in‚ 126.77734375' proc.memory.availableMB‚ 738.1328125 5†”2’- _wandbruntime‚3531  _runtime‚3531ăC*É“Ã ' lr-AdamW/pg1‚0.00013341137619415032 ' lr-AdamW/pg2‚0.00013341137619415032 trainer/global_step‚4499 % _timestamp‚1.7884200313542836e+09 _runtime‚3531.091807338 _step‚365ÂììÍÎ”È ' lr-AdamW/pg1‚0.00013341137619415032 ' lr-AdamW/pg2‚0.00013341137619415032 trainer/global_step‚4499 % _timestamp‚1.7884200313542836e+09 _runtime‚3531.091807338 _step‚365í÷ôìŽ2•- _wandbruntime‚3531  _runtime‚3531&%ÔÝ'–×O \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257498493406962 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59416176 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65121932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757531518464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288549721772544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84724983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264324650626048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚4499 % _timestamp‚1.7884200313549676e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚3531.092675186 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3744356393070 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37990958 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚366 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284607464491520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37990956 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149866953 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928}f|â'—ÜO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84745021 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149846913 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84724983 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65121932 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260566462745088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3757531518464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚288549721772544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84724983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264324650626048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 trainer/global_step‚4499 % _timestamp‚1.7884200313549676e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264324650626048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942233340928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65121932 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59416624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _runtime‚3531.092675186 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758187880960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757531518464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3744356393070 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84745021 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264323976260608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37990958 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚366 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65121930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284607464491520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260566462745088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37990956 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264323976260608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149866953 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21425666 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942063425536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260566444742144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288549527917056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257498493406962 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59416176îu ð2˜- _wandbruntime‚3531  _runtime‚3531òDÛ:±™áM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260622815783936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264381823106048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21430136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStats0F±Ë6%Monitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884200320422986e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84763501 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253809695582208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264381148727296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758350946304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65135946 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65135948 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928mpæì&šæM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3745172828242 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758350946304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879383 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚260622797780992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84743437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260622815783936 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264381823106048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21430136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884200320422986e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65135946 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943087056384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288612036577792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288612226225664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84743437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84763501 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59428804 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚253809695582208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚264381148727296 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3758350946304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149899449 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65135946 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚257554885449810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759007322112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65135948 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263384096684544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287690025314816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291675112446976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263384096684544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65822732 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65822732 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚370 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263384096684544 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65822730 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287690025314816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚291674931174400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3784537294722 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263384114687488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚291675112446976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267182614008832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263384096684544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65822732 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65822732 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151473765 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38399358 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _step‚370 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65822730 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3985944218624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3799993127936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85689553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894öX3~›#ªæM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60062416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65836746 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60062865 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85689553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3786008167650 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38407524 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263440467726336 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260340255839894 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85689553 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3785353729686 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚263440467726336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260340927317218 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151526301 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256554919149568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85669489 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3799993127936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65836748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151506235 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263440449723392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3985944218624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256554902110208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batc(WO£Qh_end/active_bytes.small_pool.allocated‚ 3799993127936 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38807756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270041251757056 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270041926122496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294800603271168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290772538257920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 % _timestamp‚1.7884201052168486e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66523532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290772562198016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4027879546368 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21876051 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825372634130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38807758 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 _runtime‚3604.954535103 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60684254 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3825372634130 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263071248618894 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4028041073152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259245221547008 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86577087 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60683807 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270041926122496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153120657 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚294800417804288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66523530 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86597125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266201766629888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66523532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38807758 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259245204507648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153100617 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 _runtime‚3604.954535103 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86597125 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60684254 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3839503130112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3840159492608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3839503130112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270041251757056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944öÐ âý2º- _wandbruntime‚3605  _runtime‚36058DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826843507058 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884201058988535e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMoM:«Ü±nitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86595541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚375 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86595541 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259300108638208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4599 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66537548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3840978933760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4028901637632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86595541 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266258101665792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4028738027008 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚259300108638208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259300125677568 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚4599 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21880579 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66537546 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66537548 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86615605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153153153 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66537548 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86615605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60696503 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3826189069094 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3840978933760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3826843507058 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884201058988535e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290834212935680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270099098602496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266258119668736 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153133087 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86595541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3840322557952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270099098602496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚375 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272900563870720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884201418393502e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87503139 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4070863220224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272900563870720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 trainer/global_step‚4649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚3641.577106957 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272899889505280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293855099081216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269019400569344 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67224332 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚418 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272900563870720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61319454 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7884201418393502e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3881145298432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154747509 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61319034 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87503139 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297925962301440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4070863220224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265856619009186 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87523177 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3880488935936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272900563870720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265857290486302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3866207973538 trainer/global_step‚4649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39216156 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚3641.577106957 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269019400569344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67238348 trainer/global_step‚4649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269075753608192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22107842 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3881308363776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788420142535433e+09 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272957736350720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87541657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67238346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414¨vn²ë&ÌåM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67238348 trainer/global_step‚4649 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67238346 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154780005 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87541657 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67238348 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87521593 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297988292389888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293916749818880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22107408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3881308363776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154759939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269075753608192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4071725927424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265912339574710 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269075753608192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61331732 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22107842 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39224324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154780005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269075771611136 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3881308363776 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788420142535433e+09 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272957736350720 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297988475746304 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272957061971968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87521593 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87541657 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154759939 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67238346 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3881964739584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262045332205568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3867024408502 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39224326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ûU!à“2Í- _wandbruntime‚3642  _runtime‚3642À.\H[ÎÕ 1loss/SRqqt3/assignment_loss‚1.395196029767394 1loss/SRqqt2/detection_loss‚3.0482537109743464 .loss/FBt1/detection_loss‚3.072447717189789 /loss/FRt1/assignment_loss‚3.775213158491886 0loss/FBt3/assignment_loss‚0.7154405400156975 1loss/SRqqt3/detection_loss‚1.7792631101608276 /loss/FBt4/detection_loss‚0.7464446815716076 0loss/FRt4/assignment_loss‚1.8296811717251937 0loss/FBt1/assignment_loss‚0.5498062968254089 /loss/FBt2/detection_loss‚2.8339657458392056 %loss/total_loss‚47.79085173492162 trainer/global_step‚4649 /loss/FRt3/detection_loss‚2.3936815014481545 1loss/SRqqt1/detection_loss‚1.7695966716045919 _step‚380 /loss/FRt2/assignment_loss‚4.182629300702002 1loss/SRqqt1/assignment_loss‚1.588268305660914 1loss/FBt4/assignment_loss‚0.23199958725219974 1loss/SRqqt4/detection_loss‚1.2006827666320734 _runtime‚3642.274963388 0loss/FRt3/assignment_loss‚3.4092277085781095 /loss/FRt2/detection_loss‚2.9216590312219437 /loss/FRt4/detection_loss‚1.6159838012730081 .loss/FBt3/detection_loss‚2.722529435157776 epoch‚3 2loss/SRqqt2/assignment_loss‚2.0666028572754427 2loss/SRqqt4/assignment_loss‚1.0535485104192048 1loss/FBt2/assignment_loss‚0.28261192634009935 /loss/FRt1/detection_loss‚2.6061181687947474 % _timestamp‚1.7884201425375028e+09°èOˆÏÚ 2loss/SRqqt4/assignment_loss‚1.0535485104192048 1loss/FBt2/assignment_loss‚0.28261192634009935 /loss/FRt1/detection_loss‚2.6061181687947474 % _timestamp‚1.7884201425375028e+09 1loss/SRqqt3/assignment_loss‚1.395196029767394 1loss/SRqqt2/detection_loss‚3.0482537109743464 .loss/FBt1/detection_loss‚3.072447717189789 /loss/FRt1/assignment_loss‚3.775213158491886 0loss/FBt3/assignment_loss‚0.7154405400156975 1loss/SRqqt3/detection_loss‚1.7792631101608276 /loss/FBt4/detection_loss‚0.7464446815716076 0loss/FRt4/assignment_loss‚1.8296811717251937 0loss/FBt1/assignment_loss‚0.5498062968254089 /^”‹°Øloss/FBt2/detection_loss‚2.8339657458392056 %loss/total_loss‚47.79085173492162 trainer/global_step‚4649 /loss/FRt3/detection_loss‚2.3936815014481545 1loss/SRqqt1/detection_loss‚1.7695966716045919 _step‚380 /loss/FRt2/assignment_loss‚4.182629300702002 1loss/SRqqt1/assignment_loss‚1.588268305660914 1loss/FBt4/assignment_loss‚0.23199958725219974 1loss/SRqqt4/detection_loss‚1.2006827666320734 _runtime‚3642.274963388 0loss/FRt3/assignment_loss‚3.4092277085781095 /loss/FRt2/detection_loss‚2.9216590312219437 /loss/FRt4/detection_loss‚1.6159838012730081 .loss/FBt3/detection_loss‚2.722529435157776 epoch‚3 2loss/SRqqt2/assignment_loss‚2.0666028572754427üâIœÊЂ(:… ±ÀäÔ€Àš~ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.39' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜…lÀÑ‚(:ô ±ÀäÔÐ¥¤€ disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 269.85546875 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.sda3.out‚ 269.85546875 disk.drbd2265.out‚0* proc.memory.percent‚16.437959671020508 disk.drbd1716.out‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 737.1953125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚95.50051689147949 cpu‚25.297802932716024) disk./.usagePercent‚17.23032750040306 disk.dm-0.in‚ 4.33203125 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚256.7443313598633 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚15305.84814453125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125" proc.memory.rssMB‚ 2693.1953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚15305.84814453125 network.recv‚ 1.379301e+06 disk.rbd1.out‚ 1.83984375 disk.sda.out‚ 269.85546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 network.sent‚ 1.0185399e+07 disk.rbd1.in‚0( disk.nvme0c0n1.out‚15305.84814453125ÖÂ&’Ò‚(:‡ ÀÀäÔܹÿ} gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W _p~Ó‚(:ó ÀÀäÔÛ¿€ disk.sda3.in‚ 126.77734375& disk.nvme0n1.out‚15349.84423828125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.rbd1.out‚ 1.83984375 disk.drbd1716.in‚0 disk.sda3.out‚ 270.3046875$ disk./.usageGB‚256.7665328979492 disk.dm-0.in‚ 4.33203125 disk.sda.out‚ 270.3046875" proc.memory.rssMB‚ 2693.1953125 cpu‚25.296730641636714 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0294119e+07* disk./.usagePercent‚17.231817464252337 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 1.385854e+06 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125( proc.memory.availableMB‚ 737.44140625* proc.memory.percent‚16.437959671020508 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 270.3046875( disk.nvme0c0n1.out‚15349.84423828125$ memory_percent‚95.49901485443115 disk.drbd2080.out‚0# disk.dm-0.out‚15349.84423828125 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda2.out‚0ˆ…˜`Ô‚(:’ ÏÀäÔÝŠ¼~ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.429' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.085800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉðÒ±}Õ‚(:ò ÏÀäԸŬ€ disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.437959671020508 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 270.72265625 disk.drbd2266.out‚0 disk.sda.out‚ 270.72265625 disk.drbd1716.in‚0& disk.nvme0n1.out‚15401.62353515625 proc.cpu.threads‚17 network.recv‚ 1.389273e+06 disk.rbd1.out‚ 1.83984375 disk.nvme0c0n1.in‚ 4.33203125 network.sent‚ 1.0299436e+07 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚95.49903869628906# disk.dm-0.out‚15401.62353515625 disk.sda1.out‚0 disk.sda3.out‚ 270.72265625 cpu‚25.295725910761618 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚15401.62353515625 disk.sda1.in‚0 disk.sda2.in‚0$ proc.memory.availableMB‚737.4375 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2693.1953125) disk./.usagePercent‚17.23263208022629% disk./.usageGB‚256.77867126464844 disk.drbd2080.in‚0 disk.drbd2266.in‚0BÒ2Ö- _wandbruntime‚3678  _runtime‚3678TÒnÆ×À & lr-AdamW/pg1‚0.0001293704501556894 & lr-AdamW/pg2‚0.0001293704501556894 trainer/global_step‚4699 $ _timestamp‚1.788420178867421e+09 _runtime‚3678.605018239 _step‚381”C22ËØÅ & lr-AdamW/pg1‚0.0001293704501556894 & lr-AdamW/pg2‚0.0001293704501556894 trainer/global_step‚4699 $ _timestamp‚1.788420178867421e+09 _runtime‚3678.605018239 _step‚381ý§Ó_¼2Ù- _wandbruntime‚3678  _runtime‚3678þžšqÜ'ÚÖO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61954018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3907697750702 $ _timestamp‚1.788420178868116e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264735634603008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚4699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚382 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271837070514688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4113700653056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67925130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271837052511744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558W¾á'ÛÛO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268642660876594 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275758527253504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271837052511744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39624558 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88449229 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67925132 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3921474741760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61954018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301051312677376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301051500241408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3922131104256 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3907697750702 $ _timestamp‚1.788420178868116e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271837070514688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264735634603008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚4699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67925132 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275758527253504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275759201618944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275759201618944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156374361 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88429191 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚382 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88449229 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156354321 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271837070514688 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4113700653056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268643332353710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67925130 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67925130 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271837052511744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944þëA”q2Ü- _wandbruntime‚3679  _runtime‚3679çÉâR¢ÝáM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301114013855232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsM͵äjEonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275816374098944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271893423553536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22333925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264790521694208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88447645 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚4699 _runtime‚3679.298571987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752u ?#ì&ÞæM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39632726 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296999286702080 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67939148 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156406857 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61967089 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271893423553536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301114013855232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3922950545408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67939146 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275816374098944 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚296999262761984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚275816374098944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271893423553536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156386791 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268699052919442 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22333925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264790538733568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156386791 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264790521694208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271893405550592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61966649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88467709 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88447645 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4114727153152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3908514185874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88447645 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚4699 _runtime‚3679.298571987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3922294169600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39632724 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E¸Éçà _runtime‚3715.450392235 _step‚385 ' lr-AdamW/pg1‚0.00012833622824547184 ' lr-AdamW/pg2‚0.00012833622824547184 trainer/global_step‚4749 % _timestamp‚1.7884202157128336e+09X0ÎèÈ ' lr-AdamW/pg1‚0.00012833622824547184 ' lr-AdamW/pg2‚0.00012833622824547184 trainer/global_step‚4749 % _timestamp‚1.7884202157128336e+09 _runtime‚3715.450392235 _step‚385Ægƒ°2é- _wandbruntime‚3715  _runtime‚3715Tú¬0Ý'ê×O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22556241 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271428702744002 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 _step‚386 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚278617165001728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157981173 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300020172847616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271429374221118 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _runtime‚3715.451315036 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3947878652354 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚4749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274654722457088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274654704454144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40032958 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300020148907520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89375281 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156807825920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68625930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62589197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884202157135637e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267480841131008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89355243 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278617839367168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274654704454144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68625932 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚278617839367168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚274654704454144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68625932 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68625930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304176795206656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278617165001728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3947878652354 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚274654722457088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4156646299136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚4749 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3963116910080 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274654722457088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274654704454144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40032958 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158001213 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 262745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89355243 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278617839367168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304176980673536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300020148907520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89375281 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3962460547584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4156807825920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68625930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62589197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884202157135637e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267480824091648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844‚*âˆY2ì- _wandbruntime‚3716  _runtime‚37166ˆRç&íáM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89373697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22561155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40041124 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 % _timestamp‚1.7884202164013412e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62602281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 trainer/global_step‚4749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963279975424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963936351232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274711057492992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3716.139450955)O*ì&îæM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3963936351232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3949349525282 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89373697 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963279975424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963936351232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948695087318 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62601824 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304239490468864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274711057492992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158033709 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚3716.139450955 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89373697 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22560700 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278674337468416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304239290335232 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300081823585280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3963279975424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158033709 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22561155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89393761 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89393761 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40041124 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274711075495936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40041126 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 % _timestamp‚1.7884202164013412e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278675011847168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62602281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267535745261568 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300081799645184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278675011847168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158013643 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158013643 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157666883584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68639948 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274711057492992 trainer/global_step‚4749 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68639948 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ƒ^•2ï- _wandbruntime‚3716  _runtime‚3716‹)­]ð× 2loss/SRqqt3/assignment_loss‚1.7856336157768964 1loss/FBt1/assignment_loss‚0.09651249705348164 1loss/SRqqt4/detection_loss‚1.1088349719842274 /loss/FBt4/detection_loss‚0.5805357548957452 _runtime‚3716.140523469 0loss/FRt4/assignment_loss‚1.5064912089487401 /loss/FRt2/detection_loss‚2.8189038617270334 1loss/SRqqt2/detection_loss‚2.2051723217591643 /loss/FBt1/detection_loss‚3.2108411490917206 /loss/FBt2/detection_loss‚3.6158860325813293 0loss/FRt2/assignment_loss‚4.1901772669383455 %loss/total_loss‚46.23091441276215 _step‚388 0loss/FRt3/assignment_loss‚3.3045617291482827 0loss/FBt2/assignment_loss‚0.2660710662081682 0loss/FBt3/assignment_loss‚0.2935378367546946 .loss/FRt3/detection_loss‚2.027223596633491 /loss/FBt3/detection_loss‚2.3865633249282836 trainer/global_step‚4749 1loss/SRqqt2/assignment_loss‚2.648273540660739 /loss/FRt4/detection_loss‚1.9504044820077535 1loss/SRqqt1/detection_loss‚1.7867079892983804 epoch‚3 0loss/FRt1/assignment_loss‚3.6563059421295816 0loss/SRqqt1/assignment_loss‚1.58205349299197 1loss/FBt4/assignment_loss‚0.22460156985976654 2loss/SRqqt4/assignment_loss‚1.0544168649034367 /loss/FRt1/detection_loss‚2.3022168215284955 1loss/SRqqt3/detection_loss‚1.6289874749524254 % _timestamp‚1.7884202164030938e+09Õ¥bñÜ 1loss/FBt1/assignment_loss‚0.09651249705348164 1loss/SRqqt4/detection_loss‚1.1088349719842274 /loss/FBt4/detection_loss‚0.5805357548957452 _runtime‚3716.140523469 0loss/FRt4/assignment_loss‚1.5064912089487401 /loss/FRt2/detection_loss‚2.8189038617270334 1loss/SRqqt2/detection_loss‚2.2051723217591643 /loss/FBt1/detection_loss‚3.2108411490917206 /loss/FBt2/detection_loss‚3.6158860325813293 0loss/FRt2/assignment_loss‚4.1901772669383455 %loss/total_loss‚46.23091441276215 _step‚388 0loss/FRt3/assignment_loss‚3.3045617291482827 0loss/FBt2/assignment_loss‚0.2660710662081682 0loss/FBt3/assignment_loss‚0.2935378367546946 .loss/FRt3/detection_loss‚2.027223596633491 /loss/FBt3/detection_loss‚2.3865633249282836 trainer/global_step‚4749 1loss/SRqqt2/assignment_loss‚2.648273540660739 /loss/FRt4/detection_loss‚1.9504044820077535 1loss/SRqqt1/detection_loss‚1.7867079892983804 epoch‚3 0loss/FRt1/assignment_loss‚3.6563059421295816 0loss/SRqqt1/assignment_loss‚1.58205349299197 1loss/FBt4/assignment_loss‚0.22460156985976654 2loss/SRqqt4/assignment_loss‚1.0544168649034367 /loss/FRt1/detection_loss‚2.3022168215284955 1loss/SRqqt3/detection_loss‚1.6289874749524254 % _timestamp‚1.7884202164030938e+09 2loss/SRqqt3/assignment_loss‚1.7856336157768964„ôú”àœò‚(:‘ üÀäÔ„«œ~ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.193' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚35.83860000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(­ó‚(:ô üÀäÔ¡«­€" proc.memory.rssMB‚ 2693.1953125 network.sent‚ 1.0431423e+07 disk.sda3.out‚ 273.45703125 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 4.33203125) disk./.usagePercent‚17.23457261733583 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.sda.in‚ 126.77734375$ disk./.usageGB‚256.8075866699219 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0 disk.sda1.out‚0$ memory_percent‚95.50197124481201* proc.memory.percent‚16.437959671020508 network.recv‚ 1.425527e+06 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚15551.45361328125( proc.memory.availableMB‚ 736.95703125 cpu‚25.29277401414224 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0# disk.dm-0.out‚15551.45361328125( disk.nvme0c0n1.out‚15551.45361328125 disk.drbd2080.out‚0 disk.dm-1.out‚ 273.45703125 disk.sda.out‚ 273.45703125 proc.cpu.threads‚17 disk.rbd0.out‚0Z#Ø’ô‚(:‡ ‹ÁäÔ´ë•~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»û¼ÑÀõ‚(:ë ‹ÁäÔËžŠ€ disk.drbd2265.in‚0 disk.drbd2266.out‚0# memory_percent‚95.5019474029541# disk.dm-0.out‚15611.08642578125 disk.drbd1716.out‚0 disk.nvme0c0n1.9ÑPþ¶in‚ 4.33203125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 736.9609375 disk.drbd1716.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚15611.08642578125 disk.sda3.out‚ 274.09375 cpu‚25.291743221464127 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.83984375 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.0539254e+07 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0* proc.memory.percent‚16.437959671020508& disk.nvme0n1.out‚15611.08642578125 disk.dm-1.out‚ 274.09375" proc.memory.rssMB‚ 2693.1953125 proc.cpu.threads‚17$ disk./.usageGB‚256.8333206176758 disk.sda.out‚ 274.09375 network.recv‚ 1.430904e+06* disk./.usagePercent‚17.236299644161814 disk.rbd0.out‚0ÏSX’ö‚(:‡ šÁäÔèÝ~ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©,sÉ÷‚(:ô šÁäÔõ‡ø disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 274.69140625 cpu‚25.290784936279504$ disk./.usageGB‚256.8268623352051 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.out‚ 1.83984375 disk.sda2.in‚0 network.sent‚ 1.0544926e+07 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.sda1.out‚0& disk.nvme0n1.out‚15658.56982421875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 274.69140625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2693.1953125 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 736.9296875* proc.memory.percent‚16.437959671020508 network.recv‚ 1.434538e+06 disk.drbd2265.in‚0 disk.sda.out‚ 274.69140625 disk.nvme0c0n1.in‚ 4.33203125* disk./.usagePercent‚17.235866223406347 disk.sda.in‚ 126.77734375 disk.sda1.in‚0$ memory_percent‚95.50213813781738( disk.nvme0c0n1.out‚15658.56982421875# disk.dm-0.out‚15658.56982421875 disk.rbd1.in‚0Wzq˜2ø- _wandbruntime‚3751  _runtime‚3751Ù!ÙYÉùà ' lr-AdamW/pg1‚0.00012729283827565263 ' lr-AdamW/pg2‚0.00012729283827565263 trainer/global_step‚4799 % _timestamp‚1.7884202520580146e+09 _runtime‚3751.795518437 _step‚389M´€ÎúÈ ' lr-AdamW/pg1‚0.00012729283827565263 ' lr-AdamW/pg2‚0.00012729283827565263 trainer/global_step‚4799 % _timestamp‚1.7884202520580146e+09 _runtime‚3751.795518437 _step‚389…# èÒ2û- _wandbruntime‚3751  _runtime‚3751Š¼dÜ'üÖO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788420252058666e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4004102715904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69326732 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307302499436032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚3751.796380865 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 trainer/global_step‚4799 _step‚390 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69326730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40441356 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4199789705216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303102709730816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270226047659008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63225733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096€ ®á'ýÛO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 trainer/global_step‚4799 _step‚390 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90281295 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3989368429518 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69326730 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159608025 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63225299 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40441356 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4004102715904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚277472356396544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4199789705216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90301333 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277472374399488 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307302311872000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303102709730816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚270226047659008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚270226030619648 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277472356396544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274215416088526 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281475802749952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63225733 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 $ _timestamp‚1.788420252058666e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4004102715904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90301333 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22784375 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3988713991762 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277472374399488 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40441358 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281475802749952 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69326732 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4003446353408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4199626081280 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274214744611410 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90281295 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159628065 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4003446353408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307302499436032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69326732 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚3751.796380865†ψã;2þ- _wandbruntime‚3752  _runtime‚3752~LÂ7$ÿáM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274271136654258 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90299749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884202527612648e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69340748 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277528709435392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281532975216640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69340748 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3752.499279431 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3752.499279431 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159660561 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63237945 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚277528709435392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚307365014519296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274271136654258 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90299749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚274270465176934 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69340746 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4200654050816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884202527612648e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40449526 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69340748 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚277528709435392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281532975216640 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40449524 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4200488343040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159640495 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307364824871424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90319813 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚277528727438336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 262745088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69340748 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281533649595392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚391 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63238380 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3989530426726 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4004922157056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22788421 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159660561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22788854 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4004922157056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚277528727438336 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990184864690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚270280951789568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚4835 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91343005 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 _runtime‚3826.290637794 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚307239674025984 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4051842312704 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚311489299563520 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚281362090245120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚278023803484678 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4037416733790 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161717823 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4052335142912 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚64237712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚311489438847488 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91343005 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚273986877665280 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4249625537536 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚285414443390976 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚70374818 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚273986894704640 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚70374818 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4052335142912 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚285413932557824 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _step‚393 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚23175596 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64237866 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚278024311438430(q2ÂT FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4249625537536 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚285414443390976 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4249740881408 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚70374818 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚154 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚273986894704640 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115343872 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚70374818 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚70374816 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4052335142912 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4036925819398 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚285413932557824 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _step‚393 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚23175596 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚64237866 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚278024311438430 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚70374816 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚281362090245120 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.788420326552715e+09 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚4835 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91343005 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41062268 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚23175444 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 161717823 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚281362108248064 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 _runtime‚3826.290637794 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 336208384 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚307239674025984 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4051842312704 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4051842312704 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚311489299563520 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚281362108248064 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚281362090245120 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚278023803484678 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4037416733790 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 161717823 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4052335142912 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚64237712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚311489438847488 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91343005 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚273986877665280 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚285413932557824 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161701844 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚91327028‰f=ò2‘- _wandbruntime‚3827  _runtime‚3827I¼{ÙN)’ÈR WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚274022188464128 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚70385622 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91334330 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚278059968807326 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚64248764 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 336208384 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _step‚394 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 161735931 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚285451127713280 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚70385624 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚64248918 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚152 % _timestamp‚1.7884203274926178e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚91350307 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41068774 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚70385624 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91350307 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚70385622 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚154 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161719952 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 4052700269568 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115343872 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4249990664192 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4052700269568 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4037289428806 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚311527679707648 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚311527540423680 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4037780343198 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚285451638546432 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4249990664192 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4052700269568 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4037289428806 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚311527679707648 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚311527540423680 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 4037780343198 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚307277573699584 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4250106008064 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0Š++U2”- _wandbruntime‚3839  _runtime‚3839wך绕ö¦ =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9155450463294983 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02261904999613762 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4992363154888153 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5931318998336792 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_2_of_2‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13804319500923157 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5327280163764954 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5535212159156799 @&Purity/0FRt4SRqqt1FBt/event_pÚŸ'¸ùroportion‚6.200397183420137e-05 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.499675452709198 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚46.41689626067529 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5651242136955261 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09191350638866425 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014323508366942406 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08805743604898453 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49035710096359253 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3331368565559387 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974581599235535 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3104924261569977 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11070948094129562 +jet/accuracy_3_of_5‚0.10171530395746231 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3698200285434723 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14393939077854156 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12222947925329208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17299270629882812 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2372387945652008 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09146182239055634 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43113914132118225 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5625704526901245 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4598890244960785 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08967792242765427 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19716157019138336 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.04732510447502136 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_7_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19881878793239594 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0972222313284874 *jet/accuracy_1_of_6‚0.9615384340286255 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111111521720886 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911717772483826 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.0960175022482872 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01686176285147667 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4778752624988556 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.3605991303920746 particle/accuracy_3_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6060605645179749 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43344929814338684 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17224407196044922 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_3‚0.01893259584903717 particle/accuracy_2_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9869468808174133 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38742560148239136 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24760203063488007 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008822419680655003 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_4_of_6‚0.03846153989434242 trainer/global_step‚4835 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.36723592877388 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_1_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976622462272644 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11127874255180359 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5150319933891296 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 particle/accuracy_4_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5239598155021667 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26453039050102234 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6359699964523315 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9758533239364624 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10646901279687881 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9402738809585571 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21790991723537445 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35736411809921265 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5404762029647827 particle/accuracy_1_of_6‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10329028218984604 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5336890816688538 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07397661358118057 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1834467202425003 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5581328272819519 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5378026366233826 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36759254336357117 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3403274118900299 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41337719559669495 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946490526199341 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8206008076667786 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06656806170940399 +jet/accuracy_3_of_4‚0.05685041844844818 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22915494441986084 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3545140027999878 particle/accuracy_5_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1922893524169922 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41696691513061523 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16213354468345642 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895516037940979 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025435367599129677 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 *jet/accuracy_2_of_6‚0.5192307829856873 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.500900387763977 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.607758641242981 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (jet/accuracy_1_of_1‚0.49595907330513 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2386256456375122 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5396842360496521 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07386244088411331 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8379837274551392 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.981793999671936 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5578075051307678 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5714809894561768 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.03279569745063782 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5495197772979736 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45402297377586365 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5763739943504333 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9852781295776367 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11780042946338654 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9894760251045227 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_5‚0.4348694682121277 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10992907732725143 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46934863924980164 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03672248497605324 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14639639854431152 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚3 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5156180262565613 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13850882649421692 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12369701266288757 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5271478295326233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5187807679176331 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5396544933319092 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5277952551841736 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2525861859321594 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_4_of_4‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14371676743030548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9631944298744202 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9787218570709229 particle/accuracy_3_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3549382984638214 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5382769107818604 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9534818530082703 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111640930176 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1313524693250656 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5274811387062073 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02807926945388317 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5511873364448547 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5533697605133057 particle/accuracy_2_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0972222313284874 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4897717535495758 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2832454442977905 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08429119735956192 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006897203624248505 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.08779257535934448 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10054877400398254 -jet/accuracy_4_of_4‚0.0027557318098843098 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33541664481163025 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6144619584083557 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1913580298423767 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3793964087963104 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49087515473365784 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9522657990455627 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5378026366233826 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11997520178556442 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_4_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11833968758583069 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5232992768287659 particle/accuracy_1_of_7‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.502112090587616 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8724490404129028 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2373044639825821 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5951388478279114 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5763888955116272 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.264814168214798 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46808287501335144 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3402778208255768 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961209893226624 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863731861114502 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28242769837379456 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5359982848167419 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.012578616850078106 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 jet/accuracy_5_of_7‚0 % _timestamp‚1.7884203401812954e+09 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5707334876060486 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416567325592 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47309374809265137 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.394400954246521 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9878485798835754 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17198148369789124 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013168816454708576 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9839054346084595 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5074636936187744 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16103488206863403 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 _step‚395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4692544937133789 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35085269808769226 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004068154376000166 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17779076099395752 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.2851170003414154 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13390472531318665 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9839946627616882 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4831660985946655 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09425287693738937 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12401964515447617 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02785966359078884 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5425956845283508 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43962711095809937 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5131650567054749 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.569444477558136 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030790116637945175 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14371676743030548 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9374999403953552 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6003140211105347 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.439334154€è¿žÉ1290283 jet/accuracy_2_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 jet/accuracy_3_of_7‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783870577812195 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48212745785713196 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12761425971984863 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4968191087245941 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41085270047187805 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23479008674621582 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6703603267669678 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.545634388923645 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44562289118766785 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7607620358467102 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1487169712781906 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1507968008518219 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 3839.92115079 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31641075015068054 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931568503379822 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07992654293775558 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18948377668857574 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881092309951782 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5799106955528259 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1ëJ¬)q–û¦ ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006897203624248505 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.08779257535934448 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10054877400398254 -jet/accuracy_4_of_4‚0.0027557318098843098 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33541664481163025 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6144619584083557 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1913580298423767 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3793964087963104 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49087515473365784 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9522657990455627 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5378026366233826 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11997520178556442 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 particle/accuracy_4_of_7‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11833968758583069 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5232992768287659 particle/accuracy_1_of_7‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.502112090587616 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8724490404129028 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2373044639825821 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9114583730697632 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5951388478279114 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6003085970878601 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5763888955116272 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.264814168214798 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46808287501335144 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3402778208255768 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961209893226624 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9863731861114502 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.28242769837379456 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5359982848167419 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.012578616850078106 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 jet/accuracy_5_of_7‚0 % _timestamp‚1.7884203401812954e+09 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5707334876060486 ."Purity/1FRt1SRqqt2FBt/event_purity‚0.03125 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1510416567325592 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47309374809265137 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.394400954246521 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9878485798835754 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17198148369789124 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013168816454708576 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9839054346084595 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5074636936187744 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16103488206863403 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 _step‚395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4692544937133789 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35085269808769226 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004068154376000166 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17779076099395752 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.2851170003414154 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13390472531318665 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9839946627616882 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4831660985946655 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.09425287693738937 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12401964515447617 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.02785966359078884 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5425956845283508 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43962711095809937 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5131650567054749 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.569444477558136 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.030790116637945175 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14371676743030548 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9374999403953552 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6003140211105347 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4393341541290283 jet/accuracy_2_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 jet/accuracy_3_of_7‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9783870577812195 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48212745785713196 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12761425971984863 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4968191087245941 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.41085270047187805 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9066193699836731 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23479008674621582 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6703603267669678 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.545634388923645 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.44562289118766785 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7607620358467102 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1487169712781906 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1507968008518219 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 _runtime‚ 3839.92115079 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31641075015068054 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9931568503379822 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07992654293775558 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18948377668857574 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881092309951782 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5799106955528259 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9155450463294983 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.20000000298023224 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.920634925365448 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02261904999613762 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4992363154888153 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5931318998336792 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_2_of_2‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13804319500923157 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5327280163764954 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.36419758200645447 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5535212159156799 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.499675452709198 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚46.41689626067529 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5651242136955261 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09191350638866425 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014323508366942406 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08805743604898453 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49035710096359253 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3331368565559387 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9974581599235535 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1515151411294937 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3104924261569977 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11070948094129562 +jet/accuracy_3_of_5‚0.10171530395746231 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3698200285434723 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14393939077854156 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12222947925329208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17299270629882812 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2372387945652008 particle/accuracy_1_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09146182239055634 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.43113914132118225 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5625704526901245 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4598890244960785 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08967792242765427 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19716157019138336 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_7‚1 particle/accuracy_7_of_7‚1 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.04732510447502136 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_7_of_7‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19881878793239594 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0972222313284874 *jet/accuracy_1_of_6‚0.9615384340286255 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111111521720886 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911717772483826 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.0960175022482872 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01686176285147667 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4778752624988556 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_4‚0.3605991303920746 particle/accuracy_3_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6060605645179749 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43344929814338684 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17224407196044922 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_3_of_3‚0.01893259584903717 particle/accuracy_2_of_7‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9869468808174133 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.38742560148239136 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24760203063488007 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008822419680655003 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_4_of_6‚0.03846153989434242 trainer/global_step‚4835 7"Purity/*FRt0SRqqt*FBt/event_purity‚0.36723592877388 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 jet/accuracy_1_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976622462272644 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11127874255180359 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_2_of_6‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5150319933891296 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 particle/accuracy_4_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5239598155021667 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26453039050102234 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6359699964523315 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9758533239364624 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10646901279687881 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9402738809585571 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21790991723537445 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35736411809921265 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5404762029647827 particle/accuracy_1_of_6‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10329028218984604 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5336890816688538 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.07397661358118057 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1834467202425003 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5581328272819519 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5378026366233826 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.36759254336357117 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.47333332896232605 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3403274118900299 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.41337719559669495 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9946490526199341 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8206008076667786 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06656806170940399 +jet/accuracy_3_of_4‚0.05685041844844818 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22915494441986084 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3545140027999878 particle/accuracy_5_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.21875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1922893524169922 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41696691513061523 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16213354468345642 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9895516037940979 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025435367599129677 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233¨¶/kY"759880066 *jet/accuracy_2_of_6‚0.5192307829856873 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.500900387763977 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.607758641242981 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 (jet/accuracy_1_of_1‚0.49595907330513 particle/accuracy_1_of_2‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2386256456375122 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5396842360496521 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07386244088411331 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8379837274551392 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5533333420753479 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.981793999671936 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5578075051307678 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5714809894561768 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_4_of_5‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666865348816 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.03279569745063782 particle/accuracy_1_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5495197772979736 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45402297377586365 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5763739943504333 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9852781295776367 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11780042946338654 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9894760251045227 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_5‚0.4348694682121277 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10992907732725143 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46934863924980164 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03672248497605324 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14639639854431152 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 epoch‚3 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5156180262565613 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13850882649421692 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12369701266288757 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6363636255264282 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5271478295326233 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5187807679176331 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5396544933319092 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5277952551841736 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2525861859321594 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_4_of_4‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14371676743030548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9631944298744202 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9787218570709229 particle/accuracy_3_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3549382984638214 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5382769107818604 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9534818530082703 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4861111640930176 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1313524693250656 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5274811387062073 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02807926945388317 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5511873364448547 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5533697605133057 particle/accuracy_2_of_5‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0972222313284874 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4897717535495758 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2832454442977905 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.08429119735956192 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632‹i,v‘—‚(:† ôÁäÔ৶~ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚118.942' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙPœäz˜‚(:ï ôÁäÔÝ›ƒ disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 1.98828125 disk.sda.in‚ 126.77734375% disk./.usageGB‚256.94619369506836( disk.nvme0c0n1.out‚15980.81005859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚15980.81005859375 disk.nvme0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 278.296875 disk.sda1.in‚0$ memory_percent‚95.98782062530518* disk./.usagePercent‚17.243874651092455 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.in‚0& disk.nvme0n1.out‚15980.81005859375) proc.memory.percent‚16.92070960998535 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 278.296875 disk.rbd1.in‚0 disk.sda.out‚ 278.296875 disk.sda1.out‚0" proc.memory.rssMB‚ 2772.2890625 cpu‚25.36431352063701 network.recv‚ 1.489217e+06 disk.drbd2265.out‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 657.35546875 network.sent‚ 1.0696124e+07 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125É4ü<2™- _wandbruntime‚3853  _runtime‚3853K)Òíɚà ' lr-AdamW/pg2‚0.00012624049768713766 trainer/global_step‚4849 % _timestamp‚1.7884203540498073e+09 _runtime‚3853.787367398 _step‚396 ' lr-AdamW/pg1‚0.00012624049768713766AQÐÎ›È _runtime‚3853.787367398 _step‚396 ' lr-AdamW/pg1‚0.00012624049768713766 ' lr-AdamW/pg2‚0.00012624049768713766 trainer/global_step‚4849 % _timestamp‚1.7884203540498073e+09ŒU­Wã2œ- _wandbruntime‚3853  _runtime‚3853wø×ÓÝ'×O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70708308 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4067452129280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91669310 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚312837616643584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312837429079552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64556286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41259510 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91689348 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4266035194880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70708310 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4266198818816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162377618 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308571417824768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 % _timestamp‚1.7884203540505192e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286677752282624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282610318156288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚312837429079552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64556286 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4052473831252 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41259510 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91689348 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4266035194880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70708310 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚282610300153344 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70708308 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4266198818816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162377618 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚308571417824768 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚4849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91689348 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282610318156288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70708310 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚308571393884672 % _timestamp‚1.7884203540505192e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41259508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162397658 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0éîÖ2Ÿ- _wandbruntime‚3854  _runtime‚3854T½Ìc áM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚308633068562432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4267057970688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70722324 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4068927932928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70722326 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279302938392168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275249665165312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275249648125952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚3854.715505482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41267678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23301277 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266894360064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312899938982400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70722326 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚275249665165312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275249648125952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4053944704180 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4068271557120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚3854.715505482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162430154 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162410088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279303609869492 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚312900126533120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41267678 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41267676 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64568953 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23301277 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282666671195136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4266894360064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4849 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚282666653192192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚312899938982400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4068271557120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23301717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4068927932928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚282666653192192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚282666671195136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚398 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70722326 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285427952095744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282033931171168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108437935104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315963254932992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277939967562752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289536390030848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71409108 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92595362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41667910 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282033259694052 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚429 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65191030 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277939950523392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285427952095744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285427970098688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚282033931171168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4108437935104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164024510 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315963254932992 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚289537064396288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4109094297600 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884203916216276e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 trainer/global_step‚4899 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4094125605624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚3891.359688019 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316025574136320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚402 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92633880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2í|¸é ³æM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164036940 trainer/global_step‚4899 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4094125605624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285484305134592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41676078 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚316025763784192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4309992630784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4109913738752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71423124 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚277994871693312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65203621 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚3891.359688019 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4109257362944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277994854653952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164057006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282089651736900 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164036940 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23527980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚282088980259576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71423126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65204058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285484323137536 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289593562497536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316025574136320 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _step‚402 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285484305134592 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92633880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92613816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚311715605445632 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/nêr9Ium_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71423124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41676076 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884203916216276e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92633880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4310158338560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944’‘+À2´- _wandbruntime‚3891  _runtime‚3891H$¡mXµÒ 1loss/SRqqt2/detection_loss‚2.4983018040657043 _runtime‚3891.360777935 1loss/SRqqt2/assignment_loss‚2.180611653205676 1loss/FBt3/assignment_loss‚0.10137582650005632 1loss/FBt4/assignment_loss‚0.15057365376735213 .loss/FRt4/detection_loss‚1.931116367672004 1loss/SRqqt1/detection_loss‚1.7420274288046593 1loss/SRqqt3/detection_loss‚1.6255319923162461 /loss/FBt1/detection_loss‚2.3746375176641674 .loss/FBt2/detection_loss‚3.575254806450435 epoch‚4 % _timestamp‚1.7884203916233037e+09 2loss/SRqqt3/assignment_loss‚1.6140942895784973 2loss/SRqqt4/assignment_loss‚1.1966316833371116 /loss/FBt4/detection_loss‚0.8541254528487722 _step‚403 1loss/SRqqt4/detection_loss‚1.0723031992994008 $loss/total_loss‚46.8453286508718 /loss/FRt1/assignment_loss‚3.225843247345516 /loss/FRt4/assignment_loss‚1.633626152487362 1loss/FBt1/assignment_loss‚0.48832902270886636 .loss/FBt3/detection_loss‚2.560815611055919 trainer/global_step‚4899 /loss/FRt2/assignment_loss‚4.515395501080682 2loss/SRqqt1/assignment_loss‚1.3163918092408602 .loss/FRt2/detection_loss‚3.089808974195929 /loss/FRt3/detection_loss‚2.3484377702293187 0loss/FRt3/assignment_loss‚3.1177415355392126 0loss/FBt2/assignment_loss‚1.0766054737780613 .loss/FRt1/detection_loss‚2.555747877699988 ?]]¶× 2loss/SRqqt4/assignment_loss‚1.1966316833371116 /loss/FBt4/detection_loss‚0.8541254528487722 _step‚403 1loss/SRqqt4/detection_loss‚1.0723031992994008 $loss/total_loss‚46.8453286508718 /loss/FRt1/assignment_loss‚3.225843247345516 /loss/FRt4/assignment_loss‚1.633626152487362 1loss/FBt1/assignment_loss‚0.48832902270886636 .loss/FBt3/detection_loss‚2.560815611055919 trainer/global_step‚4899 /loss/FRt2/assignment_loss‚4.515395501080682 2loss/SRqqt1/assignment_loss‚1.3163918092408602 .loss/FRt2/detection_loss‚3.089808974195929 /loss/FRt3/detection_loss‚2.3484377702293187 0loss/FRt3/assignment_loss‚3.1177415355392126 0loss/FBt2/assignment_loss‚1.0766054737780613 .loss/FRt1/detection_loss‚2.555747877699988 1loss/SRqqt2/detection_loss‚2.4983018040657043 _runtime‚3891.360777935 1loss/SRqqt2/assignment_loss‚2.180611653205676 1loss/FBt3/assignment_loss‚0.10137582650005632 1loss/FBt4/assignment_loss‚0.15057365376735213 .loss/FRt4/detection_loss‚1.931116367672004 1loss/SRqqt1/detection_loss‚1.7420274288046593 1loss/SRqqt3/detection_loss‚1.6255319923162461 /loss/FBt1/detection_loss‚2.3746375176641674 .loss/FBt2/detection_loss‚3.575254806450435 epoch‚4 % _timestamp‚1.7884203916233037e+09 2loss/SRqqt3/assignment_loss‚1.6140942895784973“8Û'B’·‚(:‡ °ÂäÔý”Ž~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ5pB…¸‚(:ú °ÂäÔ²’¨€ disk.drbd1716.out‚0$ memory_percent‚90.67652225494385* proc.memory.percent‚16.915202140808105) disk./.usagePercent‚17.24954009782519 disk.sda1.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1527.55859375 network.recv‚ 1.510063e+06 disk.rbd1.out‚ 2.08349609375 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0# disk.dm-0.out‚17126.82373046875 disk.sda1.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 280.44921875 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚17126.82373046875 disk.sda2.in‚0 disk.dm-0.in‚ 4.33203125 network.sent‚ 1.0961792e+07% disk./.usageGB‚257.03061294555664 disk.drbd2266.in‚0& disk.nvme0n1.out‚17126.82373046875 disk.sda2.out‚0 disk.drbd2080.in‚0 cpu‚25.35773682022902 disk.dm-1.out‚ 280.44921875 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 4.33203125# proc.memory.rssMB‚ 2771.38671875 disk.rbd1.in‚0 disk.sda3.out‚ 280.44921875 proc.cpu.threads‚17µ ‚ý’¹‚(:‡ ¿ÂäÔý‰«~ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.224' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BâB„º‚(:ù ¿ÂäÔ«û·€) disk./.usagePercent‚17.25081219926283 disk.sda2.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚17184.45947265625 disk.sda1.out‚0 disk.dm-1.out‚ 280.94921875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 cpu‚25.35656388268511 network.sent‚ 1.1069848e+07% disk./.usageGB‚257.04956817626953 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 1.515465e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0( proc.memory.availableMB‚ 1527.6640625# proc.memory.rssMB‚ 2771.38671875 disk.sda3.out‚ 280.94921875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 280.94921875 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚17184.45947265625$ memory_percent‚90.67587852478027 disk.drbd2266.out‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚17184.45947265625 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125* proc.memory.percent‚16.915202140808105 disk.rbd1.out‚ 2.08349609375 proc.cpu.threads‚17=ãÿj2»- _wandbruntime‚3927  _runtime‚3927ŸÀY‘ǼÁ & lr-AdamW/pg1‚0.0001241098436984182 & lr-AdamW/pg2‚0.0001241098436984182 trainer/global_step‚4949 % _timestamp‚1.7884204278444479e+09 _runtime‚3927.581983087 _step‚404‡'Ì½Æ & lr-AdamW/pg2‚0.0001241098436984182 trainer/global_step‚4949 % _timestamp‚1.7884204278444479e+09 _runtime‚3927.581983087 _step‚404 & lr-AdamW/pg1‚0.0001241098436984182”¡zTµ2¾- _wandbruntime‚3927  _runtime‚3927ÃèWÀÝ'¿×O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4352374525440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4134144510068 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292395702144512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149423740928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23750447 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72109908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4150080103424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109910 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292395027779072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93541452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288245622041088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93521414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚3927.582910337 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72109910 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314736467651072 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292395027779072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72109908 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4949 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72109910 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292395027779072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93521414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93541452 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284819973038576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚288245622041088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93521414 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚3927.582910337 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72109910 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314736467651072 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4149423740928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319088866116608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319088684844032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292395702144512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4149423740928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165651362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65827192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42076308•¢$Ÿ2Á- _wandbruntime‚3928  _runtime‚3928÷)lÓ‘ÂáM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4150243168768 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 KEDeviceStatsMonitor.on_train_b79P–Vatch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165663792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72123924 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72123926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93539868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚4949 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65839928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 _runtime‚3928.267920844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165663792 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93559932 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165663792 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280740078221312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚288301975079936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72123924 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚288301957076992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72123926 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4353236871168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314798118388736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165683858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93539868 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72123926 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚4949 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65839928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319151187454976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42084478 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 _runtime‚3928.267920844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65839489 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319151379200000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165683858 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280740061181952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93539868 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4150243168768 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72810710 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚409 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72810710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322214579147776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287605343428868 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291063273983488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4190409546752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3964.310738757 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4174979849476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94447466 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884204645730493e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283430363579392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167278214 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 trainer/global_step‚4999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 O‹ÃøžIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283430380618752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521ü–Upâ'ÒÜO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚291063273983488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4190409546752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚3964.310738757 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4174979849476 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94447466 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291063273983488 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66463329 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66463778 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚317819028474368 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94467504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291063255980544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94467504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72810708 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 % _timestamp‚1.7884204645730493e+09 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283430363579392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167278214 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287606014905984 trainer/global_step‚4999 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚295254339892736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283430380618752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72810708 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42484710 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚295253665527296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395391243776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4190409546752 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291063255980544 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4191065909248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317819004534272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚295253665527296 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72810710 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _step‚409 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72810710 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322214579147776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287605343428868™`–On2Ó- _wandbruntime‚3964  _runtime‚3964¼Bæ&ÔàM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4396416798720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94485984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42492878 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291119627022336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚4999 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295310837993984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4191885350400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23983069 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3964.995964046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23983526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167310710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72824726 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72824726 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72824724 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94485984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66476404 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66475945 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94485984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167290644 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42492878 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4176450722404 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4191885350400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42492876 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322277096010752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291119627022336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚317880655271936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚295311512372736 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚4999 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94465920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287661735471716 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚295310837993984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4191885350400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291119609019392 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4191228974592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291119627022336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167310710 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23983069 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283485284749312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚3964.995964046 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23983526 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430šüä±2Ö- _wandbruntime‚3964  _runtime‚39641gÔñ×× 0loss/FBt1/assignment_loss‚0.4367093156761257 /loss/FBt4/detection_loss‚0.5947518311155846 /loss/FRt1/assignment_loss‚4.213437178196052 2loss/SRqqt1/assignment_loss‚1.5396499744681424 2loss/SRqqt4/assignment_loss‚1.0645346550182218 1loss/SRqqt3/detection_loss‚1.9461503831239848 0loss/FRt2/assignment_loss‚3.0135129160351224 0loss/FBt2/assignment_loss‚0.5125463755830424 /loss/FRt2/detection_loss‚2.9812792716202914 /loss/FRt3/detection_loss‚2.2665540645731257 0loss/SRqqt1/detection_loss‚2.161329057674075 $ _timestamp‚1.788420465259771e+09 /loss/FRt4/detection_loss‚1.7859530071304603 /loss/FBt1/detection_loss‚2.4001776799559593 .loss/FBt2/detection_loss‚2.571504220366478 epoch‚4 2loss/SRqqt2/assignment_loss‚2.3202906452840373 2loss/SRqqt3/assignment_loss‚1.8502191342413425 1loss/FBt4/assignment_loss‚0.16264487093591204 /loss/FRt1/detection_loss‚2.3751861170316353 1loss/SRqqt2/detection_loss‚2.7180581833085706 _runtime‚ 3964.99719594 trainer/global_step‚4999 0loss/FRt4/assignment_loss‚1.5695861794731834 1loss/FBt3/assignment_loss‚0.26238612333933514 1loss/SRqqt4/detection_loss‚1.0680487218593795 /loss/FBt3/detection_loss‚2.9944275617599487 &loss/total_loss‚46.0987005328646ˆcWC55 _step‚411 0loss/FRt3/assignment_loss‚3.2897630650946437tlàpbØÜ 2loss/SRqqt2/assignment_loss‚2.3202906452840373 2loss/SRqqt3/assignment_loss‚1.8502191342413425 1loss/FBt4/assignment_loss‚0.16264487093591204 /loss/FRt1/detection_loss‚2.3751861170316353 1loss/SRqqt2/detection_loss‚2.7180581833085706 _runtime‚ 3964.99719594 trainer/global_step‚4999 0loss/FRt4/assignment_loss‚1.5695861794731834 1loss/FBt3/assignment_loss‚0.26238612333933514 1loss/SRqqt4/detection_loss‚1.0680487218593795 /loss/FBt3/detection_loss‚2.9944275617599487 &loss/total_loss‚46.098700532864655 _step‚411 0loss/FRt3/assignment_loss‚3.2897630650946437 0loss/FBt1/assignment_loss‚0.4367093156761257 /loss/FBt4/detection_loss‚0.5947518311155846 /loss/FRt1/assignment_loss‚4.213437178196052 2loss/SRqqt1/assignment_loss‚1.5396499744681424 2loss/SRqqt4/assignment_loss‚1.0645346550182218 1loss/SRqqt3/detection_loss‚1.9461503831239848 0loss/FRt2/assignment_loss‚3.0135129160351224 0loss/FBt2/assignment_loss‚0.5125463755830424 /loss/FRt2/detection_loss‚2.9812792716202914 /loss/FRt3/detection_loss‚2.2665540645731257 0loss/SRqqt1/detection_loss‚2.161329057674075 $ _timestamp‚1.788420465259771e+09 /loss/FRt4/detection_loss‚1.7859530071304603 /loss/FBt1/detection_loss‚2.4001776799559593 .loss/FBt2/detection_loss‚2.571504220366478 epoch‚4›)‹7ò’Ù‚(:‡ ûÂäÔ¯ç«~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÇqÚ‚(:æ ûÂäÔجÁ€ disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 283.3203125 disk.sda2.in‚0 disk.drbd2265.in‚0 cpu‚25.35196572112718 disk.sda.in‚ 126.77734375$ disk./.usageGB‚257.2691955566406 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0$ memory_percent‚90.67673683166504# proc.memory.rssMB‚ 2771.38671875 network.sent‚ 1.119606e+07 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 283.3203125 disk.nvme0n1.in‚ 4.33203125$ disk.nvme0c0n1.out‚ 17366.1953125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.recv‚ 1.533437e+06 disk.drbd2266.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 1527.5234375 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 2.146484375 disk.dm-0.out‚ 17366.1953125" disk.nvme0n1.out‚ 17366.1953125 disk.drbd2080.out‚0* proc.memory.percent‚16.915202140808105 proc.cpu.threads‚17) disk./.usagePercent‚17.26555157703918 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 283.3203125 disk.sda1.out‚0Ia=3’Û‚(:‡ ŠÃäÔ…ž€~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P+]xÜ‚(:í ŠÃäÔˆÆü disk.rbd0.out‚0) disk./.usagePercent‚17.26907910485348 disk.dm-0.out‚17406.54296875 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 2.146484375 disk.sda.out‚ 283.77734375 disk.sda1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125# disk.nvme0n1.out‚17406.54296875 disk.rbd0.in‚0 disk.sda2.out‚0 network.recv‚ 1.539855e+06 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda3.out‚ 283.77734375( proc.memory.availableMB‚ 1052.9765625# proc.memory.rssMB‚ 2771.38671875 cpu‚25.350907283877795 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 network.sent‚ 1.1304723e+07 disk.dm-0.in‚ 4.33203125 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.915202140808105 proc.cpu.threads‚17$ disk./.usageGB‚257.3217582702637 disk.dm-1.out‚ 283.77734375% disk.nvme0c0n1.out‚17406.54296875$ memory_percent‚93.57314109802246 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd1.in‚0JÉë2Ý- _wandbruntime‚4001  _runtime‚4001!HÏÀÈÞ _runtime‚4001.357207531 _step‚412 ' lr-AdamW/pg1‚0.00012194604228717027 ' lr-AdamW/pg2‚0.00012194604228717027 trainer/global_step‚5049 $ _timestamp‚1.788420501619697e+09dŽåIÍßÇ _runtime‚4001.357207531 _step‚412 ' lr-AdamW/pg1‚0.00012194604228717027 ' lr-AdamW/pg2‚0.00012194604228717027 trainer/global_step‚5049 $ _timestamp‚1.788420501619697e+09œZÜ4ç2à- _wandbruntime‚4001  _runtime‚4001ª!Ý'á×O F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231395352576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4215815188884 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95393556 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293880925925888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4232051715072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298112303275520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293880907922944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73511510 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4438557362688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚5049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325340098780160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4216469626640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231395352576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73511510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚320901541417472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95373518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42893108 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ÿâ'âÜO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73511510 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4438725180928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298112977640960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚320901541417472 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95373518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42893110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42893108 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24207290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286175587146752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231395352576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4215815188884 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73511508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95393556 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚413 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293880925925888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4232051715072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293880925925888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67100400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298112303275520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293880907922944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73511510 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298112303275520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4438557362688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚5049 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286175570107392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168905066 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325340290538496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325340098780160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4232051715072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320901565357568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290391385296276 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4216469626640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231395352576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944^· ›2ã- _wandbruntime‚4002  _runtime‚4002öWËüäáM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884205023279493e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4232871156224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4439420153856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95391972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24211853 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚5049 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328H|÷—ì&åæM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168937562 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73525526 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95412036 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4232871156224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168937562 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293937260961792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232214780416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320963216095232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286230474237952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4216631623848 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4439420153856 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293937260961792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95391972 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67112694 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24211853 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168917496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚5049 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4439581667328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298170150120960 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73525524 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24211418 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884205023279493e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944žÂ.¦D2æ- _wandbruntime‚4002  _runtime‚4002ÕófRçÌ /loss/FRt3/assignment_loss‚3.029160832365354 2loss/SRqqt3/assignment_loss‚1.5815832651595572 /loss/FBt4/assignment_loss‚0.236836413226536 0loss/SRqqt1/detection_loss‚2.443489179453429 0loss/SRqqt3/detection_loss‚1.403765775374512 .loss/FBt3/detection_loss‚3.034589833683438 -loss/FBt1/detection_loss‚2.57796710729599 $ _timestamp‚1.788420502329733e+09 _step‚415 0loss/SRqqt4/detection_loss‚0.953844846992553 /loss/FRt2/assignment_loss‚3.438797309543147 2loss/SRqqt1/assignment_loss‚1.3691425332251717 2loss/SRqqt2/assignment_loss‚1.5921369756798487 0loss/FBt2/assignment_loss‚0.3491942660883069 /loss/FRt4/detection_loss‚1.6022887202600637 /loss/FRt1/assignment_loss‚4.035274963168537 /loss/FRt4/assignment_loss‚1.220299251874288 1loss/FBt3/assignment_loss‚0.44207011921207595 1loss/SRqqt2/detection_loss‚2.6023875249398722 /loss/FRt2/detection_loss‚2.7036668586008474 _runtime‚4002.067183192 2loss/SRqqt4/assignment_loss‚1.2105176555865174 /loss/FBt2/detection_loss‚2.6820193588733674 .loss/FBt4/detection_loss‚0.730717327225169 %loss/total_loss‚44.71913681992814 epoch‚4 .loss/FRt1/detection_loss‚2.697420036091524 0loss/FBt1/assignment_loss‚0.6074311822652817 /loss/FRt3/detection_loss‚2.1745354837427535 trainer/global_step‚5049y›WèÑ 0loss/FBt2/assignment_loss‚0.3491942660883069 /loss/FRt4/detection_loss‚1.6022887202600637 /loss/FRt1/assignment_loss‚4.035274963168537 /loss/FRt4/assignment_loss‚1.220299251874288 1loss/FBt3/assignment_loss‚0.44207011921207595 1loss/SRqqt2/detection_loss‚2.6023875249398722 /loss/FRt2/detection_loss‚2.7036668586008474 _runtime‚4002.067183192 2loss/SRqqt4/assignment_loss‚1.2105176555865174 /loss/FBt2/detection_loss‚2.6820193588733674 .loss/FBt4/detection_loss‚0.730717327225169 %loss/total_loss‚44.71913681992814 epoch‚4 .loss/FRt1/detection_loss‚2.697420036091524 0loss/FBt1/assignment_loss‚0.6074311822652817 /loss/FRt3/detection_loss‚2.1745354837427535 trainer/global_step‚5049 /loss/FRt3/assignment_loss‚3.029160832365354 2loss/SRqqt3/assignment_loss‚1.5815832651595572 /loss/FBt4/assignment_loss‚0.236836413226536 0loss/SRqqt1/detection_loss‚2.443489179453429 0loss/SRqqt3/detection_loss‚1.403765775374512 .loss/FBt3/detection_loss‚3.034589833683438 -loss/FBt1/detection_loss‚2.57796710729599 $ _timestamp‚1.788420502329733e+09 _step‚415 0loss/SRqqt4/detection_loss‚0.953844846992553 /loss/FRt2/assignment_loss‚3.438797309543147 2loss/SRqqt1/assignment_loss‚1.3691425332251717 2loss/SRqqt2/assignment_loss‚1.5921369756798487Ÿ`Ãí’é‚(:‡ ™ÃäÔ† ¹~ gpu.0.gpu‚48.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.091' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8182 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÊ­zê‚(:ï ™ÃäÔÞœ¯ disk.sda.in‚ 126.77734375 disk.sda1.out‚0( disk.nvme0c0n1.out‚17450.86962890625 network.recv‚ 1.544228e+06 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 cpu‚25.34976941268062 network.sent‚ 1.1310544e+07 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.33203125$ memory_percent‚93.57180595397949$ disk./.usageGB‚257.4484672546387 disk.sda.out‚ 284.578125 disk.drbd1716.out‚0& disk.nvme0n1.out‚17450.88525390625 disk.drbd2265.out‚0( disk./.usagePercent‚17.2775826511108 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.915202140808105 proc.cpu.threads‚17# disk.dm-0.out‚17450.91259765625 disk.rbd0.in‚0 disk.sda3.out‚ 284.578125( proc.memory.availableMB‚ 1053.1953125# proc.memory.rssMB‚ 2771.38671875 disk.dm-1.out‚ 284.5781254I0’ë‚(:‡ ¨ÃäÔò·ª~ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô¢†Í}ì‚(:ò ¨ÃäԥϮ€ disk.sda1.in‚0 cpu‚25.348609296531876 disk.nvme0c0n1.in‚ 4.33203125# proc.memory.rssMB‚ 2771.38671875* proc.memory.percent‚16.915202140808105$ disk./.usageGB‚257.5072250366211 disk.rbd0.in‚0 disk.sda3.out‚ 285.04296875 disk.drbd2265.out‚0% proc.memory.availableMB‚ 1052.9375 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 285.04296875* disk./.usagePercent‚17.281525935160705 disk.rbd0.out‚0 disk.rbd1.out‚ 2.146484375 disk.sda.out‚ 285.04296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚17484.2392578125 disk.drbd2266.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚17484.2392578125 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0$ memory_percent‚93.57337951660156 network.recv‚ 1.550679e+06 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 proc.cpu.threads‚17 network.sent‚ 1.1419556e+07% disk.nvme0n1.out‚17484.2392578125 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0Û÷Fs’í‚(:‡ ·ÃäÔõ‚‘~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.519' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1038 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…=Ábvî‚(:ë ·ÃäÔÕû‚€ disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚285.5625 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0) proc.memory.availableMB‚ 1052.94140625# proc.memory.rssMB‚ 2771.38671875 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 4.33203125" disk.dm-0.out‚17525.3955078125 disk.drbd2265.in‚0 disk.dm-1.out‚285.5625$ memory_percent‚93.57335567474365 disk.drbd2266.in‚0 disk.drbd2080.in‚0 cpu‚25.347572369934703 disk.nvme0n1.in‚ 4.33203125' disk.nvme0c0n1.out‚17525.3955078125 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.1424984e+07 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375% disk.nvme0n1.out‚17525.3955078125 disk.rbd0.out‚0* proc.memory.percent‚16.915202140808105 network.recv‚ 1.554527e+06 disk.rbd1.in‚0 disk.sda.out‚285.5625 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.288234868603006% disk./.usageGB‚257.60719299316406 disk.drbd1716.in‚0™Êa2ï- _wandbruntime‚4037  _runtime‚4037R—×Éðà ' lr-AdamW/pg1‚0.00012085227389660333 ' lr-AdamW/pg2‚0.00012085227389660333 trainer/global_step‚5099 % _timestamp‚1.7884205382092807e+09 _runtime‚4037.946926781 _step‚416ƒ«&ÎñÈ ' lr-AdamW/pg1‚0.00012085227389660333 ' lr-AdamW/pg2‚0.00012085227389660333 trainer/global_step‚5099 % _timestamp‚1.7884205382092807e+09 _runtime‚4037.946926781 _step‚416 €±9‡2ò- _wandbruntime‚4037  _runtime‚4037?ù|²Ý'ó×O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328465818456064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚417 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74212310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300970941023744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74212310 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296698577868288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296698577868288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288920793674752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4272381158400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296698559865344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÛÈ [âôÜO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300971615389184 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4273037520896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4257304966048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4037.947747811 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96319608 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚328465624600576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884205382100642e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74212308 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300970941023744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296698559865344 H>DeviceStatsMonitor.on_train_b …BÐ!atch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328465818456064 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚293177427163684 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚417 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170531918 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74212310 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4273037520896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300970941023744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74212310 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4272381158400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚296698577868288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24434718 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323984078300672 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296698577868288 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚288920793674752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67736676 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293178098640800 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4272381158400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300971615389184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296698559865344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170511878 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96299570 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323984102240768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74212308 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301028113490432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328528331157504 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482578179072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4038.652964157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 _step‚418 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301028787869184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256{EInì&÷æM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24439671 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96318024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24439225 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚301028787869184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170544348 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288975680765952 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4257466963256 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170564414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170544348 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4482422957056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚5099 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67748901 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293233819206532 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96318024 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170564414 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293233147729208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96338088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296754912904192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43309678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324045729038336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4273856962048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328528151995392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚301028113490432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328528331157504 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4482578179072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4038.652964157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 _step‚418¢Ié?2ø- _wandbruntime‚4038  _runtime‚4038àÀY3WùÑ 1loss/SRqqt3/assignment_loss‚1.662243651368229 .loss/FRt1/detection_loss‚2.520337101291208 /loss/FRt1/assignment_loss‚3.302473694086075 /loss/FRt2/assignment_loss‚3.908858518600464 0loss/SRqqt1/detection_loss‚1.816310436226601 .loss/FBt2/detection_loss‚2.779300808906555 1loss/SRqqt2/assignment_loss‚2.501023018360138 /loss/FRt4/detection_loss‚1.9954002953545993 /loss/FBt4/detection_loss‚0.6325889868712254 % _timestamp‚1.7884205389165292e+09 0loss/FRt3/assignment_loss‚2.2581056153999186 2loss/SRqqt4/assignment_loss‚0.9288323854130728 1loss/FBt4/assignment_loss‚0.22407035921385185 /loss/FRt2/detection_loss‚3.0625847387313843 1loss/SRqqt4/detection_loss‚1.0006563751785844 /loss/FBt1/detection_loss‚2.7221129025731767 epoch‚4 1loss/SRqqt3/detection_loss‚1.5021266308376344 trainer/global_step‚5099 _step‚419 .loss/FBt3/detection_loss‚4.008594416081905 1loss/FBt1/assignment_loss‚0.39536333908992155 0loss/FBt2/assignment_loss‚0.5086578499525786 0loss/FBt3/assignment_loss‚0.6877707400708459 0loss/FRt4/assignment_loss‚1.3872731729995373 2loss/SRqqt1/assignment_loss‚1.3368577752695527 .loss/FRt3/detection_loss‚2.222304592717369 0loss/SRqqt2/detection_loss‚2.570479395985603 %loss/total_loss‚45.93432680058003 _runtime‚4038.653975385ƒ˜§£\úÖ /loss/FRt1/assignment_loss‚3.302473694086075 /loss/FRt2/assignment_loss‚3.908858518600464 0loss/SRqqt1/detection_loss‚1.816310436226601 .loss/FBt2/detection_loss‚2.779300808906555 1loss/SRqqt2/assignment_loss‚2.501023018360138 /loss/FRt4/detection_loss‚1.9954002953545993 /loss/FBt4/detection_loss‚0.6325889868712254 % _timestamp‚1.7884205389165292e+09 0loss/FRt3/assignment_loss‚2.2581056153999186 2loss/SRqqt4/assignment_loss‚0.9288323854130728 1loss/FBt4/assignment_loss‚0.22407035921385185 /loss/FRt2/detection_loss‚3.0625847387313843 1loss/SRqqt4/detection_loss‚1.0006563751785844 /loss/FBt1/detection_loss‚2.7221129025731767 epoch‚4 1loss/SRqqt3/detection_loss‚1.5021266308376344 trainer/global_step‚5099 _step‚419 .loss/FBt3/detection_loss‚4.008594416081905 1loss/FBt1/assignment_loss‚0.39536333908992155 0loss/FBt2/assignment_loss‚0.5086578499525786 0loss/FBt3/assignment_loss‚0.6877707400708459 0loss/FRt4/assignment_loss‚1.3872731729995373 2loss/SRqqt1/assignment_loss‚1.3368577752695527 .loss/FRt3/detection_loss‚2.222304592717369 0loss/SRqqt2/detection_loss‚2.570479395985603 %loss/total_loss‚45.93432680058003 _runtime‚4038.653975385 1loss/SRqqt3/assignment_loss‚1.662243651368229 .loss/FRt1/detection_loss‚2.520337101291208£už’û‚(:‡ ÆÃäÔˆùû} gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&å½Eü‚(:ù ÆÃäÔÒž€ proc.cpu.threads‚17 network.recv‚ 1.559091e+06 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚17572.04638671875 disk.drbd1716.out‚0$ memory_percent‚93.57359409332275) disk./.usagePercent‚17.28841740197492 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17572.04638671875# disk.dm-0.out‚17572.04638671875 disk.sda.out‚ 286.08203125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1052.90234375# proc.memory.rssMB‚ 2771.38671875 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 286.08203125 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.1430947e+07 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 286.08203125 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 cpu‚25.346474596690676% disk./.usageGB‚257.60991287231445 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0* proc.memory.percent‚16.915202140808105 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 dis ü?k.sda2.out‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0Þ.PJý‚(:„ ÕÃäÔ„³”~ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07æW+€þ‚(:õ ÕÃäÔàŸœ€ disk.sda1.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2771.38671875* proc.memory.percent‚16.915202140808105 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.out‚ 2.146484375 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda.out‚ 287.2109375 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0' proc.memory.availableMB‚ 1052.609375 proc.cpu.threads‚17* disk./.usagePercent‚17.293160709569324 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0% disk./.usageGB‚257.68059158325195 disk.nvme0c0n1.in‚ 4.33203125& disk.nvme0n1.out‚17615.61669921875$ memory_percent‚93.57538223266602 cpu‚25.345326658340127 network.sent‚ 1.1539467e+07 network.recv‚ 1.565302e+06( disk.nvme0c0n1.out‚17615.61669921875 disk.sda3.out‚ 287.2109375# disk.dm-0.out‚17615.61669921875 disk.dm-1.out‚ 287.2109375 disk.rbd1.in‚0͘@¹2ÿ- _wandbruntime‚4074  _runtime‚4074Áè;ɀà ' lr-AdamW/pg1‚0.00011975089709130381 ' lr-AdamW/pg2‚0.00011975089709130381 trainer/global_step‚5149 % _timestamp‚1.7884205745881965e+09 _runtime‚4074.325744176 _step‚420¸ë†TÎÈ trainer/global_step‚5149 % _timestamp‚1.7884205745881965e+09 _runtime‚4074.325744176 _step‚420 ' lr-AdamW/pg1‚0.00011975089709130381 ' lr-AdamW/pg2‚0.00011975089709130381¤ “:<2‚- _wandbruntime‚4074  _runtime‚40749žÝ'ƒ×O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303829578771968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172138730 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97225622 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299516211807744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172158770 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚433 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4524610857984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331591417800192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚291666000202752 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43709908 trainer/global_step‚5149 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74913110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291665983163392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4314023326720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4314023326720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303830253137408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74913110 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚331591226041856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327066615183872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚295964140508208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74913110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4298140305456 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303829578771968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚291665983163392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4297485867700 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚421 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚303830253137408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299516229810688 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4524778676224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97245660 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97245660 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚431 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 trainer/global_step‚5149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4298302302664 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303887425617408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4298956740628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43718076 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331653736113664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚422 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172171200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884205752766557e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43718076 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331653736113664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4314186392064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299572582849536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74927124 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚422 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172171200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172191266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4525470192128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296019861073940 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884205752766557e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4314842767872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172191266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43718078 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24668213 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172171200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97244076 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74927126 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299572564846592 trainer/global_step‚5149 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299572582849536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4314842767872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68385847 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97244076 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303886751238656 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚291720904333312 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331653925761536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97264140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4314186392064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844¦¢­|J2ˆ- _wandbruntime‚4075  _runtime‚4075¶çÐÜ_‰Ù 0loss/SRqqt3/detection_loss‚1.542161501199007 _step‚423 0loss/FRt3/assignment_loss‚3.1586438947253757 2loss/SRqqt1/assignment_loss‚1.5315260624601728 0loss/FBt2/assignment_loss‚0.5734452446206253 /loss/FRt4/detection_loss‚1.8337620631126421 0loss/FBt4/detection_loss‚0.47072257937195344 /loss/FRt1/assignment_loss‚3.518362777928511 2loss/SRqqt2/assignment_loss‚1.9396365569985432 1loss/FBt1/assignment_loss‚0.41698953028852964 _runtime‚ 4075.01556108 0loss/FBt4/assignment_loss‚0.2381251139447329 1loss/SRqqt1/detection_loss‚1.9496860326755614 /loss/FRt3/detection_loss‚2.5400930291248693 /loss/FBt1/detection_loss‚3.1246868697079746 trainer/global_step‚5149 1loss/SRqqt4/detection_loss‚0.9936056253982002 % _timestamp‚1.7884205752781494e+09 0loss/FRt2/assignment_loss‚3.6812297063214436 0loss/FRt4/assignment_loss‚1.3421700160418237 1loss/FBt3/assignment_loss‚0.11353941803931125 .loss/FBt2/detection_loss‚2.352807333836189 /loss/FBt3/detection_loss‚2.7183205911091397 epoch‚4 /loss/FRt1/detection_loss‚2.1637532773117223 /loss/FRt2/detection_loss‚3.0433150359562466 &loss/total_loss‚44.876929549723165 2loss/SRqqt3/assignment_loss‚1.7155343934738387 2loss/SRqqt4/assignment_loss‚1.0174298696219921 1loss/SRqqt2/detection_loss‚2.8973830264547598„Áü½dŠÞ 0loss/FRt3/assignment_loss‚3.1586438947253757 2loss/SRqqt1/assignment_loss‚1.5315260624601728 0loss/FBt2/assignment_loss‚0.5734452446206253 /loss/FRt4/detection_loss‚1.8337620631126421 0loss/FBt4/detection_loss‚0.47072257937195344 /loss/FRt1/assignment_loss‚3.518362777928511 2loss/SRqqt2/assignment_loss‚1.9396365569985432 1loss/FBt1/assignment_loss‚0.41698953028852964 _runtime‚ 4075.01556108 0loss/FBt4/assignment_loss‚0.2381251139447329 1loss/SRqqt1/detection_loss‚1.9496860326755614 /loss/FRt3/detection_loss‚2.5400930291248693 /loss/FBt1/detection_loss‚3.1246868697079746 trainer/global_step‚5149 1loss/SRqqt4/detection_loss‚0.9936056253982002 % _timestamp‚1.7884205752781494e+09 0loss/FRt2/assignment_loss‚3.6812297063214436 0loss/FRt4/assignment_loss‚1.3421700160418237 1loss/FBt3/assignment_loss‚0.11353941803931125 .loss/FBt2/detection_loss‚2.352807333836189 /loss/FBt3/detection_loss‚2.7183205911091397 epoch‚4 /loss/FRt1/detection_loss‚2.1637532773117223 /loss/FRt2/detection_loss‚3.0433150359562466 &loss/total_loss‚44.876929549723165 2loss/SRqqt3/assignment_loss‚1.7155343934738387 2loss/SRqqt4/assignment_loss‚1.0174298696219921 1loss/SRqqt2/detection_loss‚2.8973830264547598 0loss/SRqqt3/detection_loss‚1.542161501199007 _step‚423§¿èP’‹‚(:‡ äÃäÔð«–~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01h·mŒ‚(:â äÃäÔ»ƒ™ disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.nvme0n1.out‚ 17666.59375 cpu‚25.344235321483715 proc.cpu.threads‚17 network.sent‚ 1.154582e+07 network.recv‚ 1.569866e+06 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.out‚ 287.66015625$ memory_percent‚93.57566833496094 disk.rbd0.out‚0" disk.nvme0c0n1.out‚ 17666.59375 disk.sda2.in‚0 disk.dm-0.out‚ 17666.59375 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0% proc.memory.availableMB‚ 1052.5625 disk.dm-1.out‚ 287.66015625# proc.memory.rssMB‚ 2771.38671875* proc.memory.percent‚16.915202140808105$ disk./.usageGB‚257.6597671508789 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 287.66015625 disk.rbd1.in‚0 disk.sda1.in‚0* disk./.usagePercent‚17.291763164439907 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375èm¬>’‚(:‡ óÃäÔߤš~ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÃŽ‚(:ö óÃäÔÞö¤& disk.nvme0n1.out‚17710.18896484375% disk./.usageGB‚257.69324111938477 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.915202140808105 disk.nvme0n1.in‚ 4.33203125 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1052.79296875 network.sent‚ 1.1653711e+07 disk.rbd0.out‚0 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2771.38671875 network.recv‚ 1.574889e+06 disk.drbd1716.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 288.0703125* disk./.usagePercent‚17.294009630552864 disk.sda.out‚ 288.0703125 disk.sda1.out‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚93.57426166534424( disk.nvme0c0n1.out‚17710.18896484375 disk.sda3.out‚ 288.0703125# disk.dm-0.out‚17710.18896484375 disk.dm-1.in‚ 126.77734375 cpu‚25.34316327525666 sÿË‚(:’ ‚ÄäÔÕö¸~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.300399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G-œÈ}‚(:ò ‚ÄäÔ¼øË€ disk.rbd1.out‚ 2.146484375' disk.nvme0c0n1.out‚17755.7880859375& proc.memory.availableMB‚ 1053.03125# proc.memory.rssMB‚ 2771.38671875 network.recv‚ 1.578682e+06 disk.nvme0n1.in‚ 4.33203125% disk.nvme0n1.out‚17755.7880859375 disk.rbd0.in‚0 cpu‚25.342048367209717) disk./.usagePercent‚17.29619439885007$ disk./.usageGB‚257.7257957458496 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 288.50390625 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0$ memory_percent‚93.57280731201172* proc.memory.percent‚16.915202140808105 network.sent‚ 1.1658811e+07 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.out‚0 proc.cpu.threads‚17" disk.dm-0.out‚17755.7880859375 disk.dm-1.out‚ 288.50390625 disk.sda.out‚ 288.50390625 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0õ»2‘- _wandbruntime‚4110  _runtime‚4110¦ƒqÔÉ’Ã ' lr-AdamW/pg1‚0.00011864214139654721 ' lr-AdamW/pg2‚0.00011864214139654721 trainer/global_step‚5199 % _timestamp‚1.7884206111654627e+09 _runtime‚4110.903027546 _step‚424µÜ·EÎ“È _runtime‚4110.903027546 _step‚424 ' lr-AdamW/pg1‚0.00011864214139654721 ' lr-AdamW/pg2‚0.00011864214139654721 trainer/global_step‚5199 % _timestamp‚1.7884206111654627e+09¨i"ºÎ2”- _wandbruntime‚4110  _runtime‚4110’"ˆÝ'•×O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75613908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69010961 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334716900024832 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306688890885632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330149176007168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118308 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75613908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚5199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330149152067072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680¿¤C6–ÜO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334717091783168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98171712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4567747957760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚425 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294411189691392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4567915776000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298750182375616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24892212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306688216520192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302333881753088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4354352770048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330149152067072 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75613910 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75613908 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338975644864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4354352770048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69010961 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334716900024832 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚306688890885632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330149176007168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98171712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69010520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118308 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75613908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 trainer/global_step‚5199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173765582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302333881753088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4355009132544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4568605991424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5199 $ _timestamp‚1.788420611844046e+09.hòaë&™åM $ _timestamp‚1.788420611844046e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4355828573696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302390234791936 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4355172197888 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330210826744832 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334779408796160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚426 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4355172197888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44126478 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚334779598444032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298805902941348 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44126476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173818118 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98190192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298805231464024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4339792080036 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173798052 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4568605991424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69023125 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98170128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75627926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75627924 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75627926 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302390234791936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302390216788992 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173798052 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306746063365632 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173818118 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5199ªOªè2š- _wandbruntime‚4111  _runtime‚4111Ò’ª]›× /loss/FRt1/assignment_loss‚3.138879200633691 0loss/FRt4/assignment_loss‚1.7059700931563522 2loss/SRqqt4/assignment_loss‚1.1116798079169508 /loss/FRt3/detection_loss‚2.4054876526987012 trainer/global_step‚5199 1loss/SRqqt1/assignment_loss‚1.278170426218015 .loss/FRt1/detection_loss‚2.076691915186084 1loss/SRqqt2/detection_loss‚2.6993512350779314 1loss/SRqqt3/detection_loss‚1.6510967392354243 /loss/FRt2/detection_loss‚3.1084697246551514 1loss/SRqqt4/detection_loss‚0.9744369834105471 &loss/total_loss‚45.388273092786804 0loss/FRt2/assignment_loss‚3.6170464913050333 2loss/SRqqt2/assignment_loss‚1.9417850449681282 1loss/FBt1/assignment_loss‚0.35509730792707866 1loss/FBt4/assignment_loss‚0.22401931522269097 % _timestamp‚1.7884206118459568e+09 2loss/SRqqt3/assignment_loss‚1.9072334873339822 0loss/FBt3/assignment_loss‚0.2227644388203771 /loss/FBt4/detection_loss‚0.6952815817820515 _step‚427 /loss/FRt3/assignment_loss‚3.191579454324462 /loss/FRt4/detection_loss‚1.7020855805639066 /loss/FBt1/detection_loss‚2.4455338617165885 .loss/FBt3/detection_loss‚3.509583036104838 _runtime‚4111.583394597 0loss/FBt2/assignment_loss‚0.4486503568827175 1loss/SRqqt1/detection_loss‚2.0080017326575406 /loss/FBt2/detection_loss‚2.9693776249885557 epoch‚4Ýì¶jbœÜ /loss/FRt1/assignment_loss‚3.138879200633691 0loss/FRt4/assignment_loss‚1.7059700931563522 2loss/SRqqt4/assignment_loss‚1.1116798079169508 /loss/FRt3/detection_loss‚2.4054876526987012 trainer/global_step‚5199 1loss/SRqqt1/assignment_loss‚1.278170426218015 .loss/FRt1/detection_loss‚2.076691915186084 1loss/SRqqt2/detection_loss‚2.6993512350779314 1loss/SRqqt3/detection_loss‚1.6510967392354243 /loss/FRt2/detection_loss‚3.1084697246551514 1loss/SRqqt4/detection_loss‚0.9744369834105471 &loss/total_loss‚45.388273092786804 0loss/FRt2/assignment_loss‚3.6170464913050333 2loss/SRqqt2/assignment_loss‚1.9417850449681282 1loss/FBt1/assignment_loss‚0.35509730792707866 1loss/FBt4/assignment_loss‚0.22401931522269097 % _timestamp‚1.7884206118459568e+09 2loss/SRqqt3/assignment_loss‚1.9072334873339822 0loss/FBt3/assignment_loss‚0.2227644388203771 /loss/FBt4/detection_loss‚0.6952815817820515 _step‚427 /loss/FRt3/assignment_loss‚3.191579454324462 /loss/FRt4/detection_loss‚1.7020855805639066 /loss/FBt1/detection_loss‚2.4455338617165885 .loss/FBt3/detection_loss‚3.509583036104838 _runtime‚4111.583394597 0loss/FBt2/assignment_loss‚0.4486503568827175 1loss/SRqqt1/detection_loss‚2.0080017326575406 /loss/FBt2/detection_loss‚2.9693776249885557 epoch‚4«:<© ’‚(:‡ ‘ÄäԣЄ~ gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øýòß`ž‚(:Õ ‘ÄäÔÛùŒ€ disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.nvme0n1.out‚ 17798.625& proc.memory.availableMB‚ 1052.96875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 126.77734375 disk.drbd2266.in‚0$ memory_percent‚93.57318878173828 proc.cpu.threads‚17 network.recv‚ 1.582599e+06) disk./.usagePercent‚17.29784078330277 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 disk.nvme0n1.in‚ 4.33203125# proc.memory.rssMB‚ 2771.38671875 disk.sda1.in‚0 disk.nvme0c0n1.out‚ 17798.625 disk.sda3.out‚ 289.15625 disk.dm-0.out‚ 17798.625* proc.memory.percent‚16.915202140808105 cpu‚25.340999354818806 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 289.15625 network.sent‚ 1.1664688e+07% disk./.usageGB‚257.75032806396484 disk.sda.out‚ 289.15625G‘SaŸ‚(:’  ÄäÔÍí€~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[…k ‚(:ö  ÄäÔ¤¨Ÿ€ network.sent‚ 1.1773612e+07$ disk./.usageGB‚257.7632141113281 disk.sda.out‚ 289.58984375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125& proc.memory.availableMB‚ 1052.96875# proc.memory.rssMB‚ 2771.38671875 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 289.58984375 disk.drbd2265.in‚0* proc.memory.percent‚16.915202140808105 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 289.58984375 disk.rbd0.out‚0 cpu‚25.340009803278264 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0* disk./.usagePercent‚17.298705576753434 disk.dm-0.in‚ 4.33203125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.589324e+06# disk.dm-0.out‚17855.26708984375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚17855.26708984375( disk.nvme0c0n1.out‚17855.26708984375 disk.rbd1.out‚ 2.146484375 disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375$ memory_percent‚93.57318878173828c™Ôì2¡- _wandbruntime‚4147  _runtime‚4147ü¬"ɢà _step‚428 ' lr-AdamW/pg1‚0.00011752623787535846 ' lr-AdamW/pg2‚0.00011752623787535846 trainer/global_step‚5249 % _timestamp‚1.7884206476354656e+09 _runtime‚4147.372981068»mmÎ£È % _timestamp‚1.7884206476354656e+09 _runtime‚4147.372981068 _step‚428 ' lr-AdamW/pg1‚0.00011752623787535846 ' lr-AdamW/pg2‚0.00011752623787535846 trainer/global_step‚5249¬ÿ32¤- _wandbruntime‚4147  _runtime‚4147Û}œå¥×O L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76314708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚4147.373822075 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_dLÎø$evice_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76314710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44526708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25120920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301536224243024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99097764 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76314708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 % _timestamp‚1.7884206476361268e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305151515692544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309546854268416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297156413258752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379810984272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305151533695488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611050384384 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333231712890368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309547528633856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305151515692544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69648079 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4395338575872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25121369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4610890954752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0þÕ`’â'¦ÜO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76314710 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333231712890368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297156396219392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309547528633856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305151515692544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69648079 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5249 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4395338575872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4379156546516 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25121369 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4610890954752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175412474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76314708 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚4147.373822075 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76314710 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44526708 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25120920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301535552765908 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301536224243024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99097764 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175392434 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76314708 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 % _timestamp‚1.7884206476361268e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4395994938368 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309546854268416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305151533695488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305151515692544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚309546854268416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337842763274752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44526710 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297156413258752 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4379810984272 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305151533695488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4611050384384­;Æ•2§- _wandbruntime‚4148  _runtime‚4148)ïBSç&¨áM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44534878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884206483366797e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76328724 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25125420 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305207886734336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337905091090944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99096180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4611915013632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380627419444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚297211300349952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚4148.074580132 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3015919448087567aµê ©æM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337905091090944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379972981480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76328726 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337905278641664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333293363628032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99096180 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4396814379520 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4611915013632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4380627419444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25125873 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚297211300349952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309604026735104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚4148.074580132 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69660296 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚297211317389312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚301591944808756 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44534876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44534878 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _step‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884206483366797e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4396158003712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175424904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175444970 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76328724 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25125420 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99116244 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305207886734336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4396814379520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128®¹îŠë2ª- _wandbruntime‚4148  _runtime‚4148mMg”[«Õ /loss/FBt4/detection_loss‚0.6455566396440069 0loss/FBt1/assignment_loss‚0.4935944253005661 /loss/FRt4/detection_loss‚1.8440753090316835 1loss/SRqqt1/detection_loss‚1.7283776448323176 1loss/SRqqt2/detection_loss‚2.4233009488351884 /loss/FRt1/assignment_loss‚3.870005817711353 2loss/SRqqt2/assignment_loss‚2.2415017584158528 2loss/SRqqt3/assignment_loss‚1.5919646056254908 .loss/FRt2/detection_loss‚2.635700277380041 /loss/FBt1/detection_loss‚1.9334192184301524 % _timestamp‚1.7884206483381894e+09 0loss/FRt3/assignment_loss‚3.5815727797950188 /loss/FRt4/assignment_loss‚2.215824966845305 1loss/SRqqt4/detection_loss‚1.2204839020268992 _step‚431 0loss/FBt4/assignment_loss‚0.1404098953554808 .loss/FRt1/detection_loss‚2.178432223200798 .loss/FBt2/detection_loss‚3.523080212729318 epoch‚4 2loss/SRqqt1/assignment_loss‚1.4457681550859258 1loss/FBt2/assignment_loss‚0.15798032056773081 1loss/SRqqt3/detection_loss‚1.8958263762143193 .loss/FBt3/detection_loss‚3.261633411049843 %loss/total_loss‚46.27147754956826 trainer/global_step‚5249 _runtime‚4148.075617228 1loss/FBt3/assignment_loss‚0.05733906292054769 /loss/FRt3/detection_loss‚2.6155735385127183 0loss/FRt2/assignment_loss‚3.5148138709970422 2loss/SRqqt4/assignment_loss‚1.0552421890606638–‘-`¬Ú epoch‚4 2loss/SRqqt1/assignment_loss‚1.4457681550859258 1loss/FBt2/assignment_loss‚0.15798032056773081 1loss/SRqqt3/detection_loss‚1.8958263762143193 .loss/FBt3/detection_loss‚3.261633411049843 %loss/total_loss‚46.27147754956826 trainer/global_step‚5249 _runtime‚4148.075617228 1loss/FBt3/assignment_loss‚0.05733906292054769 /loss/FRt3/detection_loss‚2.6155735385127183 0loss/FRt2/assignment_loss‚3.5148138709970422 2loss/SRqqt4/assignment_loss‚1.0552421890606638 /loss/FBt4/detection_loss‚0.6455566396440069 0loss/FBt1/assignment_loss‚0.4935944253005661 /loss/FRt4/detection_loss‚1.8440753090316835 1loss/SRqqt1/detection_loss‚1.7283776448323176 1loss/SRqqt2/detection_loss‚2.4233009488351884 /loss/FRt1/assignment_loss‚3.870005817711353 2loss/SRqqt2/assignment_loss‚2.2415017584158528 2loss/SRqqt3/assignment_loss‚1.5919646056254908 .loss/FRt2/detection_loss‚2.635700277380041 /loss/FBt1/detection_loss‚1.9334192184301524 % _timestamp‚1.7884206483381894e+09 0loss/FRt3/assignment_loss‚3.5815727797950188 /loss/FRt4/assignment_loss‚2.215824966845305 1loss/SRqqt4/detection_loss‚1.2204839020268992 _step‚431 0loss/FBt4/assignment_loss‚0.1404098953554808 .loss/FRt1/detection_loss‚2.178432223200798 .loss/FBt2/detection_loss‚3.523080212729318¯ïѲ’­‚(:‡ ¯ÄäÔ¹Üû} gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fY©ïƒ®‚(:ø ¯ÄäÔ飆€( disk.nvme0c0n1.out‚17896.54443359375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.33203125$ memory_percent‚93.57359409332275 cpu‚25.338852280511716 disk.rbd1.out‚ 2.146484375 disk.drbd1716.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2771.38671875 network.sent‚ 1.1778868e+07 network.recv‚ 1.592681e+06 disk.dm-0.in‚ 4.33203125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1052.90234375 proc.cpu.threads‚17) disk./.usagePercent‚17.30084375167941 disk.sda3.out‚ 290.13671875 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚17896.54443359375 disk.sda.out‚ 290.13671875 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-1.out‚ 290.13671875 disk.drbd2266.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚17896.54443359375* proc.memory.percent‚16.915202140808105 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd1716.out‚0$ disk./.usageGB‚257.7950744628906 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0ó 3’¯‚(:‡ ¾ÄäÔ©¾†~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.342' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖS¡Az°‚(:ï ¾ÄäÔÊ• network.recv‚ 1.599574e+06* disk./.usagePercent‚17.301334518122548$ disk./.usageGB‚257.8023872375488 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 290.6640625 disk.rbd0.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚17934.5517578125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375' disk.nvme0c0n1.out‚17934.5517578125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.dm-0.in‚ 4.33203125* proc.memory.percent‚16.915225982666016 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 290.6640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 290.6640625 disk.drbd2265.in‚0$ memory_percent‚93.57514381408691 disk.nvme0n1.in‚ 4.33203125% disk.nvme0n1.out‚17934.5517578125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375( proc.memory.availableMB‚ 1052.6484375! proc.memory.rssMB‚ 2771.390625 cpu‚25.33780573426012 network.sent‚ 1.1887696e+07Ÿ(U¦2±- _wandbruntime‚4183  _runtime‚4183¿¢ÞÛDzÁ & lr-AdamW/pg2‚0.0001164034190803588 trainer/global_step‚5299 % _timestamp‚1.7884206837721958e+09 _runtime‚4183.509717572 _step‚432 & lr-AdamW/pg1‚0.0001164034190803588ƒ²å–Ì³Æ _runtime‚4183.509717572 _step‚432 & lr-AdamW/pg1‚0.0001164034190803588 & lr-AdamW/pg2‚0.0001164034190803588 trainer/global_step‚5299 % _timestamp‚1.7884206837721958e+09°±þy2´- _wandbruntime‚4183  _runtime‚4183!M|òá'µÛO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340968132476416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436980744192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77015510 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44935110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70284225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436980744192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307969167634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4420646323680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653906642944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419991885924 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312405492016640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436324381696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884206837729683e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100023816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70284661 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080ZG•s¶#¶àO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44935110 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25349551 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304322266110432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70284225 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340968315846144 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307969185637888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4436980744192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307969167634944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4420646323680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4653906642944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4436324381696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304321594633316 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419991885924 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312405492016640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177019286 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25349117 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307969185637888 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307969167634944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100003778 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436324381696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884206837729683e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100023816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚336314249773568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚5299 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77015510 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7028466:ÓM01 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312405492016640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312406166382080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚340968132476416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336314225833472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4436980744192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77015510 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177039326±GõGL2·- _wandbruntime‚4184  _runtime‚4184ÐSC•ê&¸äM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654751835648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77029524 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437143809536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788420684452993e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312462664483328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5299 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4421462758852 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4420808320888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304377986676164 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336375900511232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341030822248960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77029524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4184.191118674 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44943276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77029526 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Zu-ï&¹éM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5299 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4421462758852 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341030628406784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4420808320888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177051756 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304377986676164 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437800185344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336375900511232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70296932 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25353656 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341030822248960 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100042296 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77029524 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304377315198840 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 _runtime‚4184.191118674 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4437800185344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44943276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437143809536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77029526 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4654751835648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77029524 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177071822 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299956506877952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437143809536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100022232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788420684452993e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100022232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177071822 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177051756 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4654921737728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312462664483328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128²Å 22º- _wandbruntime‚4184  _runtime‚4184rr[nT»Î /loss/FRt2/assignment_loss‚4.216537719187529 .loss/FBt2/assignment_loss‚0.66674734605391 /loss/FBt4/detection_loss‚0.5701503030750679 _runtime‚4184.192520339 .loss/FRt2/detection_loss‚3.393332279246786 /loss/FRt4/detection_loss‚1.8509569101511163 .loss/FBt2/detection_loss‚2.680904269218445 0loss/FRt3/assignment_loss‚2.4621154045572085 2loss/SRqqt3/assignment_loss‚1.5239034720624869 1loss/FBt1/assignment_loss‚0.22006877509623088 /loss/FRt1/detection_loss‚2.3226710734042255 0loss/SRqqt1/detection_loss‚2.170263583340296 0loss/SRqqt2/detection_loss‚2.454109791914622 % _timestamp‚1.7884206844550729e+09 0loss/FRt1/assignment_loss‚3.4879114573652092 0loss/FRt4/assignment_loss‚1.9019071954361937 0loss/FBt4/assignment_loss‚0.2745288855946127 /loss/FBt3/detection_loss‚2.5376964807510376 trainer/global_step‚5299 2loss/SRqqt4/assignment_loss‚1.0618099535150187 .loss/FRt3/detection_loss‚2.330630109018209 %loss/total_loss‚45.06989261736624 1loss/SRqqt2/assignment_loss‚1.706030825773875 1loss/FBt3/assignment_loss‚0.45092097531596664 0loss/SRqqt3/detection_loss‚1.674891374575404 ,loss/FBt1/detection_loss‚2.4177851785313 2loss/SRqqt1/assignment_loss‚1.6910792257545926 1loss/SRqqt4/detection_loss‚1.0029400284269026 epoch‚4 _step‚435é ΘY¼Ó 0loss/SRqqt3/detection_loss‚1.674891374575404 ,loss/FBt1/detection_loss‚2.4177851785313 2loss/SRqqt1/assignment_loss‚1.6910792257545926 1loss/SRqqt4/detection_loss‚1.0029400284269026 epoch‚4 _step‚435 /loss/FRt2/assignment_loss‚4.216537719187529 .loss/FBt2/assignment_loss‚0.66674734605391 /loss/FBt4/detection_loss‚0.5701503030750679 _runtime‚4184.192520339 .loss/FRt2/detection_loss‚3.393332279246786 /loss/FRt4/detection_loss‚1.8509569101511163 .loss/FBt2/detection_loss‚2.680904269218445 0loss/FRt3/assignment_loss‚2.4621154045572085 2loss/SRqqt3/assignment_loss‚1.5239034720624869 1loss/FBt1/assignment_loss‚0.22006877509623088 /loss/FRt1/detection_loss‚2.3226710734042255 0loss/SRqqt1/detection_loss‚2.170263583340296 0loss/SRqqt2/detection_loss‚2.454109791914622 % _timestamp‚1.7884206844550729e+09 0loss/FRt1/assignment_loss‚3.4879114573652092 0loss/FRt4/assignment_loss‚1.9019071954361937 0loss/FBt4/assignment_loss‚0.2745288855946127 /loss/FBt3/detection_loss‚2.5376964807510376 trainer/global_step‚5299 2loss/SRqqt4/assignment_loss‚1.0618099535150187 .loss/FRt3/detection_loss‚2.330630109018209 %loss/total_loss‚45.06989261736624 1loss/SRqqt2/assignment_loss‚1.706030825773875 1loss/FBt3/assignment_loss‚0.45092097531596664³~U’á’½‚(:‡ ÍÄäÔʆ~ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýCbå{¾‚(:ð ÍÄäÔ„ñþ€ disk.rbd1.in‚0! proc.memory.rssMB‚ 2771.390625 disk.drbd2080.out‚0) disk./.usagePercent‚17.30177792317185 disk.sda1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚17975.4345703125 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚257.8089942932129 disk.sda.out‚ 291.10546875 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.1893536e+07 network.recv‚ 1.603354e+06 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚17975.4345703125 disk.rbd1.out‚ 2.146484375$ memory_percent‚93.57233047485352* proc.memory.percent‚16.915225982666016 disk.sda3.out‚ 291.10546875 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.dm-1.out‚ 291.10546875' proc.memory.availableMB‚ 1053.109375 disk.rbd0.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚17975.4345703125 cpu‚25.33678268181828ù{œd’¿‚(:‡ ÜÄäÔâà“~ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚185.164' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.0328 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä0«ÿÀ‚(:ö ÜÄäÔ•™ã€ disk.drbd1716.out‚0 cpu‚25.33570843225721 disk.rbd0.out‚0# disk.dm-0.out‚18019.24169921875* proc.memory.percent‚16.915225982666016 network.recv‚ 1.610049e+06& disk.nvme0n1.out‚18019.24169921875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 291.62890625 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 291.62890625 network.sent‚ 1.2002436e+07 disk.rbd1.out‚ 2.146484375! proc.memory.rssMB‚ 2771.390625 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0$ disk./.usageGB‚257.8276824951172 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 291.62890625$ memory_percent‚93.57540607452393 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0* disk./.usagePercent‚17.303032104082096 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1052.60546875 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚18019.24169921875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 Á³’Á‚(:‡ ëÄäÔÚò§~ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.949' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5898 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍŒáe}‚(:ò ëÄäԶǚ€ network.sent‚ 1.2007876e+07% disk./.usageGB‚257.86398696899414 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375& proc.memory.availableMB‚ 1052.84375 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 2.146484375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.915225982666016 network.recv‚ 1.613669e+06) disk./.usagePercent‚17.30546852778546 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2771.390625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 292.18359375 disk.sda1.out‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚18059.71044921875 disk.sda3.out‚ 292.18359375 disk.drbd2266.out‚0 disk.sda.out‚ 292.18359375 disk.drbd1716.in‚0& disk.nvme0n1.out‚18059.71044921875 disk.nvme0c0n1.in‚ 4.33203125# memory_percent‚93.5739517211914 proc.cpu.threads‚17# disk.dm-0.out‚18059.71044921875 cpu‚25.33464594815955á†(2Ã- _wandbruntime‚4220  _runtime‚4220о{†ÇÄÁ _step‚436 & lr-AdamW/pg1‚0.0001152739190053022 & lr-AdamW/pg2‚0.0001152739190053022 trainer/global_step‚5349 % _timestamp‚1.7884207202928314e+09 _runtime‚4220.030396653¿‚ÌÅÆ trainer/global_step‚5349 % _timestamp‚1.7884207202928314e+09 _runtime‚4220.030396653 _step‚436 & lr-AdamW/pg1‚0.0001152739190053022 & lr-AdamW/pg2‚0.0001152739190053022´}¥÷2Æ- _wandbruntime‚4220  _runtime‚42203,g)BÇÛO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100949868 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚427 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70921615 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚437 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JŸain_batch_start/allocation.all.freed‚ 178646138 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70921188 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4477310187520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307107636500724 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77716308 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310786819577344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77716310 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25578105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100929830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4220.031304626 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344093831328768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339396762716672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884207202935503e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944–ò¥æ'ÈàO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70921615 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178646138 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307108307977840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚437 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚302646826314752 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302646809275392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344094010504192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4697223847424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚315264129764864 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100949868 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77716308 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310786837580288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310786819577344 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚315264804130304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178666178 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77716310 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚315264129764864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25578105 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100929830 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4220.031304626 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344093831328768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4460827225332 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310786819577344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339396762716672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339396786656768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884207202935503e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4477310187520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100949868 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4477966550016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚427 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45343508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77716308 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45343510 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944µ˜³(L2É- _wandbruntime‚4220  _runtime‚4220_­é‚ë&ÊåM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4697918755840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884207209841359e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0P¿;ËêM % _timestamp‚1.7884207209841359e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚429 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310843190619136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77730324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100968348 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4697918755840 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310843172616192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344156332210176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302701730445312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178698674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4461643660296 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4478129615360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178698674 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339458437394432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100948284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45351676 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70934275 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307164028543572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77730326 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178678608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45351678 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚339458413454336 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4462298098260 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100968348 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4698086560768 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5349 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77730326 QCDeviceStatsMonitor.on_train_bat Ïøµch_end/inactive_split_bytes.all.peak‚ 336208384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚315321976610304 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178678608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚427 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302701713405952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4478785991168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70933846 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25582170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77730324 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25582597 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307163357066248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100948284 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310843190619136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4478785991168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚315321976610304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚315321302231552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚344156523955200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414¶*ƒ“2Ì- _wandbruntime‚4220  _runtime‚4220ˆYÝÓVÍÐ /loss/FRt4/assignment_loss‚1.625271171873266 1loss/FBt2/assignment_loss‚0.48877573665231466 /loss/FRt1/detection_loss‚2.0823590729547585 /loss/FBt1/detection_loss‚2.6543913228171214 $ _timestamp‚1.788420720985839e+09 /loss/FRt2/detection_loss‚2.9212701436012023 1loss/SRqqt1/detection_loss‚2.2509048196577255 /loss/FBt2/detection_loss‚3.0120412707328796 .loss/FBt3/detection_loss‚2.686843548502241 epoch‚4 /loss/FRt1/assignment_loss‚3.546578311401865 0loss/FRt3/assignment_loss‚3.3053425728229056 2loss/SRqqt3/assignment_loss‚1.7471354257534533 2loss/SRqqt4/assignment_loss‚0.6988012091023847 .loss/FRt3/detection_loss‚2.133311342251928 .loss/FRt4/detection_loss‚1.563882778449492 0loss/SRqqt3/detection_loss‚1.721543624997139 /loss/FBt4/detection_loss‚0.6430673201294507 /loss/FRt2/assignment_loss‚4.436577097062142 /loss/SRqqt2/detection_loss‚3.04605368298033 0loss/SRqqt4/detection_loss‚1.188523764256388 trainer/global_step‚5349 _step‚439 2loss/SRqqt1/assignment_loss‚2.4079058170318604 0loss/FBt4/assignment_loss‚0.2606191050163662 2loss/SRqqt2/assignment_loss‚1.9969521475874858 &loss/total_loss‚47.100533211140416 0loss/FBt1/assignment_loss‚0.3742278002734695 0loss/FBt3/assignment_loss‚0.3081541252322495 _runtime‚4220.723271798 ÷øÐ[ÎÕ 0loss/FRt3/assignment_loss‚3.3053425728229056 2loss/SRqqt3/assignment_loss‚1.7471354257534533 2loss/SRqqt4/assignment_loss‚0.6988012091023847 .loss/FRt3/detection_loss‚2.133311342251928 .loss/FRt4/detection_loss‚1.563882778449492 0loss/SRqqt3/detection_loss‚1.721543624997139 /loss/FBt4/detection_loss‚0.6430673201294507 /loss/FRt2/assignment_loss‚4.436577097062142 /loss/SRqqt2/detection_loss‚3.04605368298033 0loss/SRqqt4/detection_loss‚1.188523764256388 trainer/global_step‚5349 _step‚439 2loss/SRqqt1/assignment_loss‚2.4079058170318604 0loss/FBt4/assignment_loss‚0.2606191050163662 2loss/SRqqt2/assignment_loss‚1.9969521475874858 &loss/total_loss‚47.100533211140416 0loss/FBt1/assignment_loss‚0.3742278002734695 0loss/FBt3/assignment_loss‚0.3081541252322495 _runtime‚4220.723271798 /loss/FRt4/assignment_loss‚1.625271171873266 1loss/FBt2/assignment_loss‚0.48877573665231466 /loss/FRt1/detection_loss‚2.0823590729547585 /loss/FBt1/detection_loss‚2.6543913228171214 $ _timestamp‚1.788420720985839e+09 /loss/FRt2/detection_loss‚2.9212701436012023 1loss/SRqqt1/detection_loss‚2.2509048196577255 /loss/FBt2/detection_loss‚3.0120412707328796 .loss/FBt3/detection_loss‚2.686843548502241 epoch‚4 /loss/FRt1/assignment_loss‚3.546578311401865·kwcˆ’Ï‚(:‡ úÄäÔÅæ•~ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.452' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýû)ˆeЂ(:Ú úÄäÔú•´€$ disk./.usageGB‚257.8782958984375 disk.dm-1.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125% disk.nvme0c0n1.out‚18092.53515625 disk.sda2.out‚0" proc.memory.availableMB‚1053.5 cpu‚25.333699026399216 network.recv‚ 1.618106e+06 disk.rbd0.in‚0 disk.sda3.out‚292.5625* proc.memory.percent‚16.915225982666016 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚18092.53515625 disk.rbd0.out‚0* disk./.usagePercent‚17.306428812046395 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125! proc.memory.rssMB‚ 2771.390625 network.sent‚ 1.2013737e+07 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚292.5625 disk.drbd1716.in‚0# memory_percent‚93.5699462890625 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚18092.53515625 disk.dm-1.out‚292.5625 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 proc.cpu.threads‚17Œñz,‘Ñ‚(:† ‰ÅäÔÔá‘~ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºP’ŽyÒ‚(:î ‰ÅäÔœ÷‰€$ disk.nvme0n1.out‚18134.681640625 disk.rbd0.out‚0 network.recv‚ 1.624469e+06& disk.nvme0c0n1.out‚18134.681640625 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0$ memory_percent‚93.57004165649414' proc.memory.availableMB‚ 1053.484375 cpu‚25.332648266052367 disk.drbd2265.out‚0 disk.sda.out‚ 293.19921875 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 network.sent‚ 1.212228e+07 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.in‚0! proc.memory.rssMB‚ 2771.390625 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 4.33203125$ disk./.usageGB‚257.9119758605957! disk.dm-0.out‚18134.681640625 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 293.19921875* disk./.usagePercent‚17.308689102566202 disk.drbd2080.out‚0 disk.dm-1.out‚ 293.19921875* proc.memory.percent‚16.915225982666016 disk.dm-1.in‚ 126.77734375ãrWû2Ó- _wandbruntime‚4256  _runtime‚4256Çx½ÉÉÔà ' lr-AdamW/pg1‚0.00011413797303631134 ' lr-AdamW/pg2‚0.00011413797303631134 trainer/global_step‚5399 % _timestamp‚1.7884207562664595e+09 _runtime‚4256.004017818 _step‚440êLÉ{ÎÕÈ trainer/global_step‚5399 % _timestamp‚1.7884207562664595e+09 _runtime‚4256.004017818 _step‚440 ' lr-AdamW/pg1‚0.00011413797303631134 ' lr-AdamW/pg2‚0.00011413797303631134¸ëü$2Ö- _wandbruntime‚4256  _runtime‚42560>8á'×ÛO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180272990 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78417108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4518295993344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347219585987072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101855882 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4740105114624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚4256.005056588 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180293030 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318122767513088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚441 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25806537 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚5399 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45751910 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4740262447104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71558870 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25806960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313604471519744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78417108 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347219404714496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4740105114624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚4256.005056588 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180293030 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318122767513088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4518952355840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚318123441878528 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318122767513088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4518952355840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚441 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128¹šGqŸ2Ù- _wandbruntime‚4256  _runtime‚4256¸ã¾ÚãM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101874336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 [MDeviceStatsMonitor.on_‰/ûætrain_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 _step‚442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25811116 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884207572018225e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342540950337536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309950070410980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309949398933656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 4256.9396821 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78431126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928“þÁî&ÛèM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25811116 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884207572018225e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78431126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101894400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342540950337536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71571619 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25811541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180325526 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101894400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309950070410980 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309949398933656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180305460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313660842561536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45760078 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71571192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4519771796992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180305460 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78431124 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4741128696320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318179939979776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚318180614358528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 4256.9396821 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305446919933952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78431126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313660824558592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45760076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4502478999704 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚318180614358528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚313660842561536 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180325526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78431124 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101874336 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4519115421184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101874336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 _step‚442 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4503133437668 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ºÞÚ@2Ü- _wandbruntime‚4256  _runtime‚4256O$ÇËZÝÔ .loss/FBt3/detection_loss‚3.050547162691752 &loss/total_loss‚45.437613118688326 0loss/SRqqt1/detection_loss‚2.020073716737786 1loss/SRqqt4/detection_loss‚0.9877034897007138 .loss/FBt2/detection_loss‚5.032923618952434 $ _timestamp‚1.788420757203305e+09 0loss/FBt3/assignment_loss‚0.4119144152694692 /loss/FRt1/detection_loss‚2.1702036236723266 2loss/SRqqt1/assignment_loss‚1.7548054106989686 2loss/SRqqt2/assignment_loss‚1.8900117527912645 1loss/SRqqt4/assignment_loss‚1.195509943920356 .loss/FRt3/detection_loss‚2.369037407967779 /loss/FRt4/detection_loss‚1.7757287716591044 0loss/FBt2/assignment_loss‚0.2557499300067623 /loss/FRt2/detection_loss‚2.7426280648477617 trainer/global_step‚5399 0loss/FRt1/assignment_loss‚3.7630987328787646 0loss/FRt3/assignment_loss‚3.0645612202308796 2loss/SRqqt3/assignment_loss‚1.4623378742712014 1loss/SRqqt3/detection_loss‚1.6634701163028225 /loss/FBt1/detection_loss‚1.8484934923740535 _runtime‚4256.940736109 /loss/FRt2/assignment_loss‚2.885395126958047 1loss/SRqqt2/detection_loss‚2.3109370750539444 _step‚443 1loss/FBt4/assignment_loss‚0.20436334577684118 0loss/FBt1/assignment_loss‚0.4199112200966248 /loss/FBt4/detection_loss‚0.6023676531590292 epoch‚4 0loss/FRt4/assignment_loss‚1.5558399526696456 ¹{v_ÞÙ trainer/global_step‚5399 0loss/FRt1/assignment_loss‚3.7630987328787646 0loss/FRt3/assignment_loss‚3.0645612202308796 2loss/SRqqt3/assignment_loss‚1.4623378742712014 1loss/SRqqt3/detection_loss‚1.6634701163028225 /loss/FBt1/detection_loss‚1.8484934923740535 _runtime‚4256.940736109 /loss/FRt2/assignment_loss‚2.885395126958047 1loss/SRqqt2/detection_loss‚2.3109370750539444 _step‚443 1loss/FBt4/assignment_loss‚0.20436334577684118 0loss/FBt1/assignment_loss‚0.4199112200966248 /loss/FBt4/detection_loss‚0.6023676531590292 epoch‚4 0loss/FRt4/assignment_loss‚1.5558399526696456 .loss/FBt3/detection_loss‚3.050547162691752 &loss/total_loss‚45.437613118688326 0loss/SRqqt1/detection_loss‚2.020073716737786 1loss/SRqqt4/detection_loss‚0.9877034897007138 .loss/FBt2/detection_loss‚5.032923618952434 $ _timestamp‚1.788420757203305e+09 0loss/FBt3/assignment_loss‚0.4119144152694692 /loss/FRt1/detection_loss‚2.1702036236723266 2loss/SRqqt1/assignment_loss‚1.7548054106989686 2loss/SRqqt2/assignment_loss‚1.8900117527912645 1loss/SRqqt4/assignment_loss‚1.195509943920356 .loss/FRt3/detection_loss‚2.369037407967779 /loss/FRt4/detection_loss‚1.7757287716591044 0loss/FBt2/assignment_loss‚0.2557499300067623 /loss/FRt2/detection_loss‚2.7426280648477617»‹¼æ’ß‚(:‡ ˜ÅäÔ³à±~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.337' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¨0”à‚(:ö ˜ÅäÔÖÕ¾€* proc.memory.percent‚16.915225982666016 network.sent‚ 1.212867e+07 disk.drbd1716.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 1053.6640625 disk.sda.out‚ 293.77734375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 293.77734375 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125# disk.dm-0.out‚18184.06005859375% disk./.usageGB‚257.91704177856445 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18184.06005859375 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 cpu‚25.331653972583688 network.recv‚ 1.629476e+06 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2771.390625* disk./.usagePercent‚17.309029080571676 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚93.56894493103027 disk.nvme0c0n1.in‚ 4.33203125 proc.cpu.threads‚17 disk.rbd1.out‚ 2.146484375 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚18184.06005859375 disk.dm-1.out‚ 293.77734375'ÇÀó’á‚(:‡ §ÅäÔ®¯ú} gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.209' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.0418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à(‘{â‚(:ð §ÅäÔíö disk.drbd2265.in‚0# disk.dm-0.out‚18310.06396484375! proc.memory.rssMB‚ 2771.390625 disk.nvme0n1.in‚ 4.33203125 disk.dm-1.out‚ 294.18359375* disk./.usagePercent‚17.310357503666847 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda.out‚ 294.18359375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 network.recv‚ 1.635036e+06 disk.sda1.in‚0& disk.nvme0n1.out‚18310.06396484375 disk.sda3.out‚ 294.18359375 cpu‚25.33067870796647 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125$ disk./.usageGB‚257.9368362426758 disk.rbd0.in‚0& proc.memory.availableMB‚ 1053.65625 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375# memory_percent‚93.5689926147461* proc.memory.percent‚16.915225982666016 network.sent‚ 1.223676e+07 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚18310.06396484375 disk.sda2.in‚0 disk.sda2.out‚0®ëÈtã‚(:’ ¶ÅäÔŸŸµ~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Š77~ä‚(:ó ¶ÅäÔ€ú½€ disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0$ memory_percent‚93.57113838195801 disk.drbd2265.in‚0 disk.sda.out‚ 294.7734375 disk.drbd1716.out‚0( proc.memory.availableMB‚ 1053.3046875 network.sent‚ 1.2242773e+07 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0 network.recv‚ 1.639499e+06$ disk./.usageGB‚257.9418029785156 disk.rbd0.out‚0# disk.dm-0.out‚18368.55224609375 disk.dm-1.out‚ 294.7734375 disk.sda.in‚ 126.77734375 disk.drbd2080.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.310690825476428 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18368.55224609375 disk.sda3.out‚ 294.7734375 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375& disk.nvme0n1.out‚18368.55224609375 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016 cpu‚25.329696352879587L“êÎ2å- _wandbruntime‚4292  _runtime‚4292} \Éæà trainer/global_step‚5449 % _timestamp‚1.7884207927799084e+09 _runtime‚4292.517414673 _step‚444 ' lr-AdamW/pg1‚0.00011299581790282362 ' lr-AdamW/pg2‚0.00011299581790282362D•‰IÎçÈ _step‚444 ' lr-AdamW/pg1‚0.00011299581790282362 ' lr-AdamW/pg2‚0.00011299581790282362 trainer/global_step‚5449 % _timestamp‚1.7884207927799084e+09 _runtime‚4292.517414673¼Îg¦°2è- _wandbruntime‚4292  _runtime‚4292¿¥yï›éÚO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312679720235540 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181919882 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350345199987200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102801972 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788420792780557e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308137239370752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102801972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316422123462144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312680391712656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320981405261312 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316422123462144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4559938161664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312680391712656 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79117910 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46160308 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320982079626752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4559281799168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350345199987200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4543152341904 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚320981405261312 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4559281799168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102801972 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79117908 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308137222331392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72196268 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4783533419520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 $ _timestamp‚1.788420792780557e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26035506 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46160310 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308137239370752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102801972 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4783363504128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605½j_è2ë- _wandbruntime‚4293  _runtime‚4293çïÚdë&ìåM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79131926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26040595 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316478494503936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4293.195089127 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345623487220736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884207934570293e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316478476500992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚308192143501312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 trainer/global_step‚5449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350407699114496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46168476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312736112278388 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316478476500992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79131924 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 8638156800ƒ¥W–"íêM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316478476500992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚350407892956672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79131926 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102820452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79131924 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79131926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26040595 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72208609 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181952378 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316478494503936 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72209073 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26040133 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚4293.195089127 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321039252106752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345623487220736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316478494503936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79131924 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884207934570293e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4560757602816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚446 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181952378 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102800388 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345623511160832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102800388 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4543314339112 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102820452 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321038577728000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316478476500992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4784211893760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3081921'â=îZ43501312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312735440801064 trainer/global_step‚5449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181932312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚350407699114496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46168476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312736112278388¾ö‚ô72î- _wandbruntime‚4293  _runtime‚4293Är«VïÐ 0loss/FRt3/assignment_loss‚3.1718168885020885 0loss/FBt1/assignment_loss‚0.6853845035657287 .loss/FRt1/detection_loss‚2.225626244493153 trainer/global_step‚5449 % _timestamp‚1.7884207934586039e+09 0loss/FRt1/assignment_loss‚3.5821822583675385 0loss/FRt2/assignment_loss‚3.3879192054271696 0loss/FBt2/assignment_loss‚0.2655054235005342 /loss/FBt2/detection_loss‚3.3719102484839305 %loss/total_loss‚46.47850217354481 epoch‚4 2loss/SRqqt2/assignment_loss‚2.1219251718251937 1loss/SRqqt3/assignment_loss‚1.535110037566887 /loss/SRqqt3/detection_loss‚1.80836242078631 0loss/FRt4/assignment_loss‚1.7631794105876575 1loss/FBt4/assignment_loss‚0.16765347652421073 /loss/FRt3/detection_loss‚2.0769719092522636 1loss/SRqqt4/detection_loss‚1.0456380046800606 _runtime‚4293.196038048 0loss/FBt3/assignment_loss‚0.2319331073997546 /loss/FRt2/detection_loss‚3.1976990938186645 .loss/FBt3/detection_loss‚3.441284492611885 _step‚447 2loss/SRqqt1/assignment_loss‚1.8025593892063783 /loss/FRt4/detection_loss‚1.9025556955825198 .loss/FBt1/detection_loss‚2.179654613137245 0loss/SRqqt4/assignment_loss‚1.22745990096074 0loss/SRqqt1/detection_loss‚2.155585662826248 0loss/SRqqt2/detection_loss‚2.442052356658443 /loss/FBt4/detection_loss‚0.6885326577802048PÕ‚¾[ðÕ 0loss/FRt3/assignment_loss‚3.1718168885020885 0loss/FBt1/assignment_loss‚0.6853845035657287 .loss/FRt1/detection_loss‚2.225626244493153 trainer/global_step‚5449 % _timestamp‚1.7884207934586039e+09 0loss/FRt1/assignment_loss‚3.5821822583675385 0loss/FRt2/assignment_loss‚3.3879192054271696 0loss/FBt2/assignment_loss‚0.2655054235005342 /loss/FBt2/detection_loss‚3.3719102484839305 %loss/total_loss‚46.47850217354481 epoch‚4 2loss/SRqqt2/assignment_loss‚2.1219251718251937 1loss/SRqqt3/assignment_loss‚1.535110037566887 /loss/SRqqt3/detection_loss‚1.80836242078631 0loss/FRt4/assignment_loss‚1.7631794105876575 1loss/FBt4/assignment_loss‚0.16765347652421073 /loss/FRt3/detection_loss‚2.0769719092522636 1loss/SRqqt4/detection_loss‚1.0456380046800606 _runtime‚4293.196038048 0loss/FBt3/assignment_loss‚0.2319331073997546 /loss/FRt2/detection_loss‚3.1976990938186645 .loss/FBt3/detection_loss‚3.441284492611885 _step‚447 2loss/SRqqt1/assignment_loss‚1.8025593892063783 /loss/FRt4/detection_loss‚1.9025556955825198 .loss/FBt1/detection_loss‚2.179654613137245 0loss/SRqqt4/assignment_loss‚1.22745990096074 0loss/SRqqt1/detection_loss‚2.155585662826248 0loss/SRqqt2/detection_loss‚2.442052356658443 /loss/FBt4/detection_loss‚0.6885326577802048¿d¾oÌ‘ñ‚(:† ÅÅäÔ³¨™~ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚126.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø’o~ò‚(:ó ÅÅäÔ’) disk./.usagePercent‚17.31158685184627 disk.drbd2265.in‚0& disk.nvme0n1.out‚18411.28271484375 disk.sda2.out‚0! proc.memory.rssMB‚ 2771.390625 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-1.out‚ 295.23828125 disk.rbd1.in‚0 disk.sda.out‚ 295.23828125 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0$ disk./.usageGB‚257.9551544189453 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚18411.28271484375 disk.sda3.out‚ 295.23828125' proc.memory.availableMB‚ 1053.234375 disk.dm-0.in‚ 4.33203125 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0# disk.dm-0.out‚18411.28271484375 cpu‚25.32866381244078 disk.drbd2266.out‚0 network.sent‚ 1.2248672e+07 network.recv‚ 1.643291e+06 disk.drbd2266.in‚0* proc.memory.percent‚16.915225982666016 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda2.in‚0$ memory_percent‚93.57156753540039º›¾9‘ó‚(:† ÔÅäÔѨÿ} gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚133.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.653 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÌË„wô‚(:ì ÔÅäÔʳ‹ disk.drbd1716.out‚0% disk.nvme0c0n1.out‚18460.68359375* disk./.usagePercent‚17.312239159043514 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.drbd1716.in‚0' proc.memory.availableMB‚ 1053.234375 proc.cpu.threads‚17 disk.dm-1.out‚ 295.76171875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 4.33203125* proc.memory.percent‚16.915225982666016 network.sent‚ 1.2357538e+07 disk.drbd2265.out‚0# disk.nvme0n1.out‚18460.68359375 disk.sda2.out‚0 disk.sda3.out‚ 295.76171875 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 295.76171875 disk.sda1.out‚0! proc.memory.rssMB‚ 2771.390625 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0$ memory_percent‚93.57156753540039 cpu‚25.327643556206596 network.recv‚ 1.650055e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.out‚18460.68359375 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚257.9648742675781 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.146484375¸ZgJ2õ- _wandbruntime‚4328  _runtime‚4328£dϱÉöà ' lr-AdamW/pg1‚0.00011184769162825689 ' lr-AdamW/pg2‚0.00011184769162825689 trainer/global_step‚5499 % _timestamp‚1.7884208290821638e+09 _runtime‚4328.819699284 _step‚448–qÙÎ÷È ' lr-AdamW/pg1‚0.00011184769162825689 ' lr-AdamW/pg2‚0.00011184769162825689 trainer/global_step‚5499 % _timestamp‚1.7884208290821638e+09 _runtime‚4328.819699284 _step‚448À:®+42ø- _wandbruntime‚4328  _runtime‚4328/>ù³ß'ùÙO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46568710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚ 4328.82062469 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600923967488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310882428859392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323840043009536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319239793407488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72833824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚5499 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583987681312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024ŸTÛ;ä'úÞO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323840043009536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183546734 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600923967488 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46568710 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚353470971193344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79818710 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103728024 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26264655 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚ 4328.82062469 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600923967488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4583333243556 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4600267604992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103707986 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚348644373366272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚353470777337856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚310882428859392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323840043009536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79818708 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323840717374976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319239793407488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72833824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚5499 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚348644397306368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583987681312 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚449 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103728024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚461 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315466433580064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319239793407488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46568708 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4826403971584 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ÁžÒÒl2û- _wandbruntime‚4329  _runtime‚4329«3LüãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 4329.49639456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26269161 KEDevxR‹iceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5499 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323897215476224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601087032832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844M3f¿î&ýèM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584804116484 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79832726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 4329.49639456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183559164 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚353533293426688 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72846037 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚348706048044032 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183579230 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26269161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5499 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103746504 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319296146446336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323897215476224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601087032832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46576876 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103726440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79832724 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79832726 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4584149678520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103746504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÂKy³2þ- _wandbruntime‚4329  _runtime‚4329VFj°YÿÓ epoch‚4 0loss/FBt2/assignment_loss‚0.5709727282768914 1loss/FBt4/assignment_loss‚0.13855052099807105 /loss/FBt1/detection_loss‚2.9669808745384216 /loss/FBt4/detection_loss‚0.6600190579382226 2loss/SRqqt1/assignment_loss‚1.5570708359865582 1loss/SRqqt2/assignment_loss‚1.615672373052301 2loss/SRqqt3/assignment_loss‚1.5309760426481565 .loss/FRt2/detection_loss‚2.895617708083122 .loss/FRt4/detection_loss‚1.775401122123003 /loss/FRt4/assignment_loss‚1.681085228919983 /loss/FBt3/detection_loss‚2.8694109405790056 _runtime‚4329.497777099 0loss/FRt2/assignment_loss‚3.6842131307048183 0loss/FRt3/assignment_loss‚3.0076227876875135 /loss/FRt1/detection_loss‚2.5376625069089838 1loss/SRqqt4/detection_loss‚1.0638788717370185 % _timestamp‚1.7884208297600482e+09 0loss/FRt1/assignment_loss‚3.5391337275505066 0loss/FBt1/assignment_loss‚0.3881634306162596 /loss/FRt3/detection_loss‚2.4100394666194918 1loss/SRqqt1/detection_loss‚2.1528995133498134 0loss/SRqqt2/detection_loss‚2.754890994778995 %loss/total_loss‚45.93155742564744 trainer/global_step‚5499 .loss/FBt2/detection_loss‚3.000149871621813 _step‚451 2loss/SRqqt4/assignment_loss‚1.0855929291112318 0loss/FBt3/assignment_loss‚0.4036168599767344 0loss/SRqqt3/detection_loss‚1.641935901840528úäF^€Ø /loss/FRt1/detection_loss‚2.5376625069089838 1loss/SRqqt4/detection_loss‚1.0638788717370185 % _timestamp‚1.7884208297600482e+09 0loss/FRt1/assignment_loss‚3.5391337275505066 0loss/FBt1/assignment_loss‚0.3881634306162596 /loss/FRt3/detection_loss‚2.4100394666194918 1loss/SRqqt1/detection_loss‚2.1528995133498134 0loss/SRqqt2/detection_loss‚2.754890994778995 %loss/total_loss‚45.93155742564744 trainer/global_step‚5499 .loss/FBt2/detection_loss‚3.000149871621813 _step‚451 2loss/SRqqt4/assignment_loss‚1.0855929291112318 0loss/FBt3/assignment_loss‚0.4036168599767344 0loss/SRqqt3/detection_loss‚1.641935901840528 epoch‚4 0loss/FBt2/assignment_loss‚0.5709727282768914 1loss/FBt4/assignment_loss‚0.13855052099807105 /loss/FBt1/detection_loss‚2.9669808745384216 /loss/FBt4/detection_loss‚0.6600190579382226 2loss/SRqqt1/assignment_loss‚1.5570708359865582 1loss/SRqqt2/assignment_loss‚1.615672373052301 2loss/SRqqt3/assignment_loss‚1.5309760426481565 .loss/FRt2/detection_loss‚2.895617708083122 .loss/FRt4/detection_loss‚1.775401122123003 /loss/FRt4/assignment_loss‚1.681085228919983 /loss/FBt3/detection_loss‚2.8694109405790056 _runtime‚4329.497777099 0loss/FRt2/assignment_loss‚3.6842131307048183 0loss/FRt3/assignment_loss‚3.0076227876875135ÃÒ}i'’‚(:‡ ãÅäÔ³’ª~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄD r‚‚(:ç ãÅäÔíݯ% proc.memory.availableMB‚ 1052.9375 cpu‚25.32662807342874 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda.in‚ 126.77734375% disk./.usageGB‚257.97995376586914 disk.sda1.out‚0# disk.nvme0n1.out‚18502.40234375$ memory_percent‚93.57337951660156 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 296.4453125 disk.sda.out‚ 296.4453125 network.recv‚ 1.653238e+06 disk.dm-0.in‚ 4.33203125 disk.dm-0.out‚18502.40234375 disk.rbd1.in‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016* disk./.usagePercent‚17.313251156826365 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2080.in‚0 disk.rbd1.out‚ 2.146484375 disk.dm-1.out‚ 296.4453125 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚18502.40234375 proc.cpu.threads‚17 network.sent‚ 1.2362869e+07 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0N~ ƒ‚(:’ òÅäÔ¨¿þ} gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚135.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.062200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•jgxw„‚(:ì òÅäÔý˜”€ disk.dm-1.out‚ 296.87890625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 296.87890625 network.recv‚ 1.660052e+06 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375( proc.memory.availableMB‚ 1052.3828125 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 2.146484375 disk.sda1.out‚0* proc.memory.percent‚16.915225982666016 cpu‚25.3256369869695 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.sda.out‚ 296.87890625 disk.rbd0.in‚0! proc.memory.rssMB‚ 2771.390625 disk.drbd2265.out‚0 disk.drbd2266.out‚0 network.sent‚ 1.2471602e+07 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0$ disk.nvme0n1.out‚18550.185546875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125& disk.nvme0c0n1.out‚18550.185546875 disk.sda2.in‚0# memory_percent‚93.5767650604248) disk./.usagePercent‚17.31458521208729 disk.dm-0.in‚ 4.33203125! disk.dm-0.out‚18550.185546875 disk.nvme0n1.in‚ 4.33203125$ disk./.usageGB‚257.9998321533203 disk.drbd2080.out‚0ê¦2…- _wandbruntime‚4364  _runtime‚4364MG°²É†Ã ' lr-AdamW/pg1‚0.00011069383348040597 ' lr-AdamW/pg2‚0.00011069383348040597 trainer/global_step‚5549 % _timestamp‚1.7884208650845332e+09 _runtime‚4364.822111349 _step‚452R±VJÎ‡È ' lr-AdamW/pg1‚0.00011069383348040597 ' lr-AdamW/pg2‚0.00011069383348040597 trainer/global_step‚5549 % _timestamp‚1.7884208650845332e+09 _runtime‚4364.822111349 _step‚452ÄhëRØ2ˆ- _wandbruntime‚4364  _runtime‚43648ž±á'‰ÛO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4364.823070769 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641253410816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73471552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104634038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351726910249472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 % _timestamp‚1.7884208650852726e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322057445349888 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104634038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73471087 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 trainer/global_step‚5549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568Çåû/ŠàO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322057445349888 QCDeviceStatsMonitor.on_train_batch_start/ÿx™Ú·$allocation.small_pool.freed‚ 104634038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4869653123584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73471087 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4641909773312 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4641909773312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4869493693952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185153546 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80519508 trainer/global_step‚5549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351726934189568 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚318251803970356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313627652426752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326699355123200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4364.823070769 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185173586 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46977110 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4641253410816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73471552 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104634038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326698680757760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351726910249472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4624168582964 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313627635387392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4624823020720 % _timestamp‚1.7884208650852726e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚356596587313152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430Å[0ïØ‹‚(:… ÆäÔ¤æÍ~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”dÅkŒ‚(:à ÆäÔ˜™Ð€ disk.rbd0.out‚0 network.recv‚ 1.66424e+06* disk./.usagePercent‚17.313546333513344 disk.drbd2265.in‚0! disk.dm-0.out‚18592.189453125 disk.sda.out‚297.5625$ disk.nvme0n1.out‚18592.189453125 disk.sda2.in‚0$ disk./.usageGB‚257.9843521118164 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚18592.189453125 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.2477528e+07 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.nvme0n1.in‚ 4.33203125$ memory_percent‚93.57547760009766 disk.sda3.out‚297.5625 disk.drbd2265.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 1052.59375 disk.dm-1.out‚297.5625 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125* proc.memory.percent‚16.915225982666016 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.rbd0.in‚0 cpu‚25.32468976878075! proc.memory.rssMB‚ 2771.390625$y™2- _wandbruntime‚4365  _runtime‚4365™ÂÜÝê&ŽäM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80533526 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642072838656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46985276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322113798388736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318308196013204 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80533526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313682539517952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4624985017928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185186016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313682556557312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26498940 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 $ _timestamp‚1.788420865763307e+09 _step‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46985278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26498940 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326756527603200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 $ _timestamp‚1.788420865763307e+09 _step‚454 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104672556 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185206082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80533524 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185186016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4642729214464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73484218 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚5549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80533526 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4642072838656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318307524535880 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80533524 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73483736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚322113780385792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46985276 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4870347010048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104652492 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚326756527603200 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356659101839360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185206082 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322113798388736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318308196013204 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928ÆÛQjó2- _wandbruntime‚4365  _runtime‚4365©g‹Ö[‘Õ 1loss/FBt4/assignment_loss‚0.21249964316635248 -loss/FRt1/detection_loss‚2.16482000797987 .loss/FRt2/detection_loss‚2.969640802952551 /loss/FBt3/detection_loss‚2.1582325994968414 trainer/global_step‚5549 2loss/SRqqt3/assignment_loss‚1.7091439793170509 /loss/FRt3/detection_loss‚2.3590935267964186 .loss/FBt2/detection_loss‚2.553359442949295 _runtime‚4365.502568559 0loss/FRt2/assignment_loss‚3.8041050703294816 2loss/SRqqt1/assignment_loss‚1.2828058007742382 1loss/SRqqt2/assignment_loss‚2.186877536986555 0loss/SRqqt2/detection_loss‚2.682023702987603 1loss/SRqqt3/detection_loss‚1.8096694585125326 % _timestamp‚1.7884208657651157e+09 /loss/FRt4/detection_loss‚1.7244302946384307 1loss/SRqqt4/detection_loss‚0.8777555820177978 2loss/SRqqt4/assignment_loss‚0.8363294767766771 1loss/FBt1/assignment_loss‚0.09663039377192036 1loss/SRqqt1/detection_loss‚2.1414930634200573 /loss/FBt4/detection_loss‚0.7961394952940224 epoch‚4 _step‚455 0loss/FRt4/assignment_loss‚1.6259153893700353 /loss/FRt1/assignment_loss‚3.808608129620552 0loss/FBt3/assignment_loss‚0.7761373301036656 /loss/FBt1/detection_loss‚2.6528441071510316 %loss/total_loss‚44.33076116828235 0loss/FRt3/assignment_loss‚2.6166513671680374 1loss/FBt2/assignment_loss‚0.48555496670132925)µÚw’Ú 0loss/FRt3/assignment_loss‚2.6166513671680374 1loss/FBt2/assignment_loss‚0.48555496670132925 1loss/FBt4/assignment_loss‚0.21249964316635248 -loss/FRt1/detection_loss‚2.16482000797987 .loss/FRt2/detection_loss‚2.969640802952551 /loss/FBt3/detection_loss‚2.1582325994968414 trainer/global_step‚5549 2loss/SRqqt3/assignment_loss‚1.7091439ÊL†¬é793170509 /loss/FRt3/detection_loss‚2.3590935267964186 .loss/FBt2/detection_loss‚2.553359442949295 _runtime‚4365.502568559 0loss/FRt2/assignment_loss‚3.8041050703294816 2loss/SRqqt1/assignment_loss‚1.2828058007742382 1loss/SRqqt2/assignment_loss‚2.186877536986555 0loss/SRqqt2/detection_loss‚2.682023702987603 1loss/SRqqt3/detection_loss‚1.8096694585125326 % _timestamp‚1.7884208657651157e+09 /loss/FRt4/detection_loss‚1.7244302946384307 1loss/SRqqt4/detection_loss‚0.8777555820177978 2loss/SRqqt4/assignment_loss‚0.8363294767766771 1loss/FBt1/assignment_loss‚0.09663039377192036 1loss/SRqqt1/detection_loss‚2.1414930634200573 /loss/FBt4/detection_loss‚0.7961394952940224 epoch‚4 _step‚455 0loss/FRt4/assignment_loss‚1.6259153893700353 /loss/FRt1/assignment_loss‚3.808608129620552 0loss/FBt3/assignment_loss‚0.7761373301036656 /loss/FBt1/detection_loss‚2.6528441071510316 %loss/total_loss‚44.33076116828235Ç61’“‚(:‡ ÆäÔ™¹«~ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò:M|”‚(:ñ ÆäÔå“¡€ disk.drbd2080.in‚0 disk.sda.out‚ 298.26953125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.2575014e+07 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125) disk./.usagePercent‚17.31761659730023 disk.drbd2080.out‚0# disk.dm-0.out‚18637.94873046875 disk.drbd2266.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2771.390625 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.146484375 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 298.26953125 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚18637.94873046875$ disk./.usageGB‚258.0450019836426 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 cpu‚25.32375903416115 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 298.26953125 disk.drbd2266.out‚0 disk.rbd0.out‚0$ memory_percent‚93.57719421386719* proc.memory.percent‚16.915225982666016 proc.cpu.threads‚17 network.recv‚ 1.670775e+06& disk.nvme0n1.out‚18637.94873046875% proc.memory.availableMB‚ 1052.3125tÙ`’•‚(:‡ ŸÆäÔ„ø¡~ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ΀q–‚(:æ ŸÆäÔñš disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17! disk./.usageGB‚258.0712890625 disk.dm-1.out‚ 298.765625 disk.sda1.out‚0 disk.rbd1.out‚ 2.146484375% disk.nvme0n1.out‚18681.3740234375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.out‚0" disk.dm-0.out‚18681.3740234375 network.sent‚ 1.2672179e+07 network.recv‚ 1.675575e+06) disk./.usagePercent‚17.31938074521868 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 298.765625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 cpu‚25.322818886357368' disk.nvme0c0n1.out‚18681.3740234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda.out‚ 298.765625! proc.memory.rssMB‚ 2771.390625 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 126.77734375$ memory_percent‚93.57566833496094% proc.memory.availableMB‚ 1052.5625* proc.memory.percent‚16.915225982666016å5q`2—- _wandbruntime‚4401  _runtime‚4401šÓ¬Ç˜Á ' lr-AdamW/pg1‚0.00010953448392157939 ' lr-AdamW/pg2‚0.00010953448392157939 trainer/global_step‚5599 % _timestamp‚1.7884209016580994e+09 _runtime‚ 4401.3956335 _step‚456EÀ‘Ì™Æ % _timestamp‚1.7884209016580994e+09 _runtime‚ 4401.3956335 _step‚456 ' lr-AdamW/pg1‚0.00010953448392157939 ' lr-AdamW/pg2‚0.00010953448392157939 trainer/global_step‚5599ÈgÁ=2š- _wandbruntime‚4401  _runtime‚4401Û) ´á'›ÛO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324875079289344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316372858954752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105580128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4682895579136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186780398 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324875079289344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74109330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316372841915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47385510 _step‚457 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4682239216640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354809471072768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4682239216640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81220310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329557992871424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚4401.396542405 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105580128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 % _timestamp‚1.7884209016588252e+09‚Pžæ'œàO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚354809471072768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4682239216640 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81220310 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4665003922372 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329557992871424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _runtime‚4401.396542405 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105560090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105580128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324875097292288 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329557318505984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321038517314880 % _timestamp‚1.7884209016588252e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105560090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81220310 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324875079289344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316372858954752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚354809447132672 trainer/global_step‚5599 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105580128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4682895579136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359722307098112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186780398 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324875079289344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47385508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74109330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329557318505984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321037845837764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316372841915392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47385510 _step‚457 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81220308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26723369 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4682239216640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936Éì÷22- _wandbruntime‚4402  _runtime‚4402áìŽr»žäM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47393678 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321094237880612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913548582400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81234326 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329614490972672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81234326 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329615165351424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186812868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5599 $ _timestamp‚1.788420902363603e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354871121810432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4402.101805224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4683715020288 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931!DÕž/ 432328192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4683715020288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4683058644480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚458 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324931450331136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81234324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324931432328192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4683715020288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321093566403288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329615165351424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4683058644480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186832934 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26728317 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359784646452736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4913703804416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚458 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74121995 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321094237880612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4913548582400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316427746045952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81234326 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329614490972672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359784825614848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324931450331136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81234326 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105578544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186832934 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186812868 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329615165351424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324931432328192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105598608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186812868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105578544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚5599 $ _timestamp‚1.788420902363603e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚354871121810432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚354871097870336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4402.101805224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74121542 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47393676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105598608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26727866ʨŠ >2 - _wandbruntime‚4402  _runtime‚4402ˆ½Y¡Ó 2loss/SRqqt1/assignment_loss‚1.5183291670612313 0loss/FBt3/assignment_loss‚0.2397736396790909 trainer/global_step‚5599 0loss/FBt2/assignment_loss‚0.5819000647064327 /loss/FRt2/detection_loss‚2.8998121227536884 .loss/FBt2/detection_loss‚2.155756273052909 /loss/FBt4/detection_loss‚0.7977054657814143 0loss/FRt1/assignment_loss‚3.2845285200491183 2loss/SRqqt3/assignment_loss‚1.5594063399152622 1loss/SRqqt2/detection_loss‚2.3586708919422046 &loss/total_loss‚44.719887785004424 _runtime‚4402.103096321 0loss/FBt1/assignment_loss‚0.4885427864889304 1loss/SRqqt1/detection_loss‚1.8195749439976432 /loss/FRt2/assignment_loss‚3.506887936592102 epoch‚4 0loss/FRt4/assignment_loss‚1.7699536112221805 2loss/SRqqt4/assignment_loss‚0.9024062144518312 1loss/FBt4/assignment_loss‚0.16622005572078508 1loss/SRqqt4/detection_loss‚1.1383244914727078 .loss/FBt1/detection_loss‚3.373524864514669 0loss/FRt3/assignment_loss‚2.7082070394566187 1loss/SRqqt2/assignment_loss‚1.939578134868596 /loss/FRt1/detection_loss‚2.2337425772736714 /loss/FRt3/detection_loss‚2.6210187650040577 -loss/FRt4/detection_loss‚2.00161739167842 0loss/SRqqt3/detection_loss‚1.613804508621494 .loss/FBt3/detection_loss‚3.040601978699366 % _timestamp‚1.7884209023656194e+09 _step‚459¼<~e^¢Ø /loss/FRt2/assignment_loss‚3.506887936592102 epoch‚4 0loss/FRt4/assignment_loss‚1.7699536112221805 2loss/SRqqt4/assignment_loss‚0.9024062144518312 1loss/FBt4/assignment_loss‚0.16622005572078508 1loss/SRqqt4/detection_loss‚1.1383244914727078 .loss/FBt1/detection_loss‚3.373524864514669 0loss/FRt3/assignment_loss‚2.7082070394566187 1loss/SRqqt2/assignment_loss‚1.939578134868596 /loss/FRt1/detection_loss‚2.2337425772736714 /loss/FRt3/detection_loss‚2.6210187650040577 -loss/FRt4/detection_loss‚2.00161739167842 0loss/SRqqt3/detection_loss‚1.613804508621494 .loss/FBt3/detection_loss‚3.040601978699366 % _timestamp‚1.7884209023656194e+09 _step‚459 2loss/SRqqt1/assignment_loss‚1.5183291670612313 0loss/FBt3/assignment_loss‚0.2397736396790909 trainer/global_step‚5599 0loss/FBt2/assignment_loss‚0.5819000647064327 /loss/FRt2/detection_loss‚2.8998121227536884 .loss/FBt2/detection_loss‚2.155756273052909 /loss/FBt4/detection_loss‚0.7977054657814143 0loss/FRt1/assignment_loss‚3.2845285200491183 2loss/SRqqt3/assignment_loss‚1.5594063399152622 1loss/SRqqt2/detection_loss‚2.3586708919422046 &loss/total_loss‚44.719887785004424 _runtime‚4402.103096321 0loss/FBt1/assignment_loss‚0.4885427864889304 1loss/SRqqt1/detection_loss‚1.8195749439976432Ë|«:ã‘£‚(:† ®ÆäÔÅß”~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.411 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(:Ï키‚(:÷ ®ÆäÔšü€% disk./.usageGB‚258.06360626220703( disk.nvme0c0n1.out‚18728.59228515625 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125& disk.nvme0n1.out‚18728.59228515625 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda.out‚ 299.51953125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1052.77734375 disk.dm-1.out‚ 299.51953125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.294921875 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375# disk.dm-0.out‚18728.59228515625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 299.51953125 cpu‚25.321909218359128 network.sent‚ 1.2677295e+07* disk./.usagePercent‚17.318865146044722 disk.drbd2265.in‚0 disk.drbd2080.in‚0$ memory_percent‚93.57435703277588! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd2266.in‚0 network.recv‚ 1.67951e+06 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 proc.cpu.threads‚17®åߥ‚(:’ ½ÆäÔÁß°~ gpu.0.gpu‚11.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.599599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÿmwy¦‚(:î ½ÆäÔí¥¢€) disk./.usagePercent‚17.32059114884057 disk.sda.in‚ 126.77734375( disk.nvme0c0n1.out‚18770.89306640625 disk.rbd1.out‚ 2.294921875 disk.sda2.out‚0 disk.sda.out‚ 300.234375 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2771.390625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 300.234375 proc.cpu.threads‚17 network.recv‚ 1.685363e+06 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 300.234375 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.915225982666016 cpu‚25.32099127057156$ disk./.usageGB‚258.0893249511719 disk.drbd2266.in‚0& disk.nvme0n1.out‚18770.89306640625 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0$ memory_percent‚93.57438087463379( proc.memory.availableMB‚ 1052.7734375 network.sent‚ 1.2785708e+07# disk.dm-0.out‚18770.89306640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0ã¾³/2§- _wandbruntime‚4437  _runtime‚4437»—ªûȨ _step‚460 ' lr-AdamW/pg1‚0.00010836988455848757 ' lr-AdamW/pg2‚0.00010836988455848757 trainer/global_step‚5649 % _timestamp‚1.7884209381096046e+09 _runtime‚ 4437.84713944&sˆÍ©Ç trainer/global_step‚5649 % _timestamp‚1.7884209381096046e+09 _runtime‚ 4437.84713944 _step‚460 ' lr-AdamW/pg1‚0.00010836988455848757 ' lr-AdamW/pg2‚0.00010836988455848757ÌaJÿQ2ª- _wandbruntime‚4437  _runtime‚4437»ùyá'«ÛO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26952054 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81921108 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332415956254208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 % _timestamp‚1.7884209381103098e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4723881384960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955827015680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106486142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _step‚461 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955990639616 trainer/global_step‚5649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362847811031552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4723881384960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872èð׬àO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚357892007955968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚332415956254208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327692749234688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26952054 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319118048443392 TDDeviceStatsMonitor.onÚzÅY_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4723225022464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81921108 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793910 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚332415956254208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4706493699536 % _timestamp‚1.7884209381103098e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4723881384960 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955827015680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106486142 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _step‚461 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚332416630619648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81921108 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74746402 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4955990639616 trainer/global_step‚5649 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319118065482752 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362847811031552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323824559182288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188407250 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327692731231744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4723881384960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440Í*浧2­- _wandbruntime‚4438  _runtime‚4438ò†é&®ãM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106504596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327749084270592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26956464 _step‚462 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956916 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81935126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 # _timestamp‚1.78842093879924e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327749102273536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332473803099648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0…ü†Mî&¯èM _runtime‚4438.537140441 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362910502542848 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319172952573952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74758994 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74758540 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4956669752832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188459786 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327749084270592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106524660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47802078 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4724044450304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 # _timestamp‚1.78842093879924e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327749102273536 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47802076 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4956843849216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327749102273536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106504596 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188439720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚332473803099648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188439720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106504596 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327749084270592 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81935124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4724700826112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26956464 _step‚462 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81935124 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188459786 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357953634753536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106524660 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26956916 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81935126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4706655696744 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75384397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48202310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182550 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚465 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4764210828288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82621910 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330510383174144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360974520899072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928ÆôÔLå'¾ßO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82621908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4764867190784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335274594002432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚330510383174144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360974520899072 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107432232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335275268367872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190054142 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335274594002432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82621908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚365973541825536 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107432232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75384397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48202310 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27182550 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚465 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27182089 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚365973731486720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4999186647552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4747329038944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚330510383174144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚330510401177088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75384860 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321863272010752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4764210828288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4764867190784 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82621910 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4999020926464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÑŸÙ¾D2¿- _wandbruntime‚4475  _runtime‚4475ª|¶âë&ÀåM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884209755096633e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚466 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335332440847872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4748145474116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75397665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190066572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚330566754215936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4765686631936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27186702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999869999104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326665650138104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4765686631936F°’ÁêM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚335332440847872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361036171636736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_trR6b Ñ$ain_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190086638 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚330566736212992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚487 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366036239672320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚466 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190086638 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107430648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335332440847872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107430648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4748145474116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75397665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚5699 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚361036195576832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190066572 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚330566754215936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4765686631936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190066572 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48210478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27186702 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4999869999104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚330566754215936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27187187 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107450712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚330566736212992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326665650138104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321918159101952 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4765686631936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75397178 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884209755096633e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚326666321615428 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107450712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4747491036152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844Òì4/G2Â- _wandbruntime‚4475  _runtime‚44757W¶VÃÐ 0loss/FBt2/assignment_loss‚0.4725984103046358 .loss/FRt1/detection_loss‚2.416049295793409 /loss/FRt3/detection_loss‚2.3498680141161787 $ _timestamp‚1.788420975511316e+09 0loss/FRt2/assignment_loss‚3.9929361180825667 /loss/FRt4/detection_loss‚1.5879703159802234 1loss/SRqqt4/detection_loss‚1.0642297513027121 ,loss/FBt3/detection_loss‚2.4259238583701 _step‚467 2loss/SRqqt2/assignment_loss‚2.2619935716352155 1loss/SRqqt3/detection_loss‚1.8966276209490995 /loss/FBt4/detection_loss‚0.6874378688197823 _runtime‚ 4475.24874033 /loss/FRt2/detection_loss‚3.2121138626878913 0loss/FRt1/assignment_loss‚2.8151496765406234 0loss/FBt3/assignment_loss‚0.7260736987393882 0loss/FBt4/assignment_loss‚0.2201719240739368 0loss/SRqqt2/detection_loss‚2.503835483904808 .loss/FBt2/detection_loss‚3.992894244194031 2loss/SRqqt3/assignment_loss‚1.3991739926859736 0loss/FRt3/assignment_loss‚2.8804574839093466 1loss/SRqqt1/assignment_loss‚1.571285661398655 2loss/SRqqt4/assignment_loss‚1.0877750795604526 1loss/SRqqt1/detection_loss‚1.8362919305052077 1loss/FBt1/assignment_loss‚0.45353198480249074 /loss/FBt1/detection_loss‚2.2506761153539023 trainer/global_step‚5699 /loss/FRt4/assignment_loss‚1.874133742772616 %loss/total_loss‚45.97919970648324 epoch‚4«\w[ÄÕ ,loss/FBt3/detection_loss‚2.4259238583701 _step‚467 2loss/SRqqt2/assignment_loss‚2.2619935716352155 1loss/SRqqt3/detection_loss‚1.8966276209490995 /loss/FBt4/detection_loss‚0.6874378688197823 _runtime‚ 4475.24874033 /loss/FRt2/detection_loss‚3.2121138626878913 0loss/FRt1/assignment_loss‚2.8151496765406234 0loss/FBt3/assignment_loss‚0.7260736987393882 0loss/FBt4/assignment_loss‚0.2201719240739368 0loss/SRqqt2/detection_loss‚2.503835483904808 .loss/FBt2/detection_loss‚3.992894244194031 2loss/SRqqt3/assignment_loss‚1.3991739926859736 0loss/FRt3/assignment_loss‚2.8804574839093466 1loss/SRqqt1/assignment_loss‚1.571285661398655 2loss/SRqqt4/assignment_loss‚1.0877750795604526 1loss/SRqqt1/detection_loss‚1.8362919305052077 1loss/FBt1/assignment_loss‚0.45353198480249074 /loss/FBt1/detection_loss‚2.2506761153539023 trainer/global_step‚5699 /loss/FRt4/assignment_loss‚1.874133742772616 %loss/total_loss‚45.97919970648324 epoch‚4 0loss/FBt2/assignment_loss‚0.4725984103046358 .loss/FRt1/detection_loss‚2.416049295793409 /loss/FRt3/detection_loss‚2.3498680141161787 $ _timestamp‚1.788420975511316e+09 0loss/FRt2/assignment_loss‚3.9929361180825667 /loss/FRt4/detection_loss‚1.5879703159802234 1loss/SRqqt4/detection_loss‚1.0642297513027121Óäk«¨’Å‚(:‡ ùÆäÔျ~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—?ª€Æ‚(:õ ùÆäÔŒ¯°$ disk./.usageGB‚258.1504020690918 disk.sda.out‚ 302.53515625 disk.nvme0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0" disk.dm-0.out‚18935.6630859375$ memory_percent‚93.57502460479736) proc.memory.availableMB‚ 1052.66796875 disk.sda2.out‚0 network.sent‚ 1.2911869e+07 network.recv‚ 1.703831e+06 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.39013671875 disk.sda1.out‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚18935.6630859375 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0* proc.memory.percent‚16.915225982666016) disk./.usagePercent‚17.32469008547129 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 302.53515625 disk.dm-0.in‚ 4.33203125 cpu‚25.317374108195782 disk.drbd2265.out‚0 disk.dm-1.out‚ 302.53515625 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚18935.6630859375 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2771.390625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17É œÇ‚(:‘ ˆÇäÔ¨¯~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.69' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.937999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û{(œzÈ‚(:ï ˆÇäÔó£ disk.drbd2265.in‚0 network.sent‚ 1.3020946e+07 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd1.out‚2.453125 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18982.91357421875# disk.dm-0.out‚18982.91357421875* proc.memory.percent‚16.915225982666016 disk.sda3.out‚ 303.078125 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 303.078125 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚18982.91357421875 disk.sda2.out‚0 disk.drbd1716.out‚0% disk./.usageGB‚258.13167572021484 disk.dm-1.out‚ 303.078125 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0$ memory_percent‚93.57507228851318 disk.dm-0.in‚ 4.33203125) proc.memory.availableMB‚ 1052.66015625 cpu‚25.316489192969033 network.recv‚ 1.710581e+06* disk./.usagePercent‚17.323433344485704 disk.rbd0.out‚0 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2771.390625 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375q­\S2É- _wandbruntime‚4511  _runtime‚4511xþËWÈÊ $ _timestamp‚1.788421011282873e+09 _runtime‚4511.020408732 _step‚468 ' lr-AdamW/pg1‚0.00010602590826602791 ' lr-AdamW/pg2‚0.00010602590826602791 trainer/global_step‚5749h³MÃÍËÇ _runtime‚4511.020408732 _step‚468 ' lr-AdamW/pg1‚0.00010602590826602791 ' lr-AdamW/pg2‚0.00010602590826602791 trainer/global_step‚5749 $ _timestamp‚1.788421011282873e+09Ôí4÷Œ2Ì- _wandbruntime‚4511  _runtime‚4511퀑”à'ÍÚO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4805196634112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108338246 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329396642917104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48610708 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324608461499392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4788164378352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324608478538752 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0]ìö5éÎßO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚364057081722368 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191660954 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5042273750528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4805196634112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76022885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27412177 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83322708 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76023360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚333328035116544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚338133906116096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108338246 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329396642917104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333328053119488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48610708 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364057057782272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚473 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5749 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108358284 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27412650 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369099149034496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324608461499392 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚338133231750656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108358284 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚369099355472896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4805196634112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48610710 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚475 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333328035116544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4788164378352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324608478538752Õ™CnÆ2Ï- _wandbruntime‚4511  _runtime‚4511šÙ×’ë&ÐåM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4511.703913268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 trainer/global_step‚5749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333384388155392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329451692005512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4788326375560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191713490 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324663382669312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚470 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5042941684736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324663365629952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884210119658213e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333384388155392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚369161846144000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324663382669312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚470 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5042941684736 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364118732460032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324663365629952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884210119658213e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4806016061952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚338191078596096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚338191078596096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333384406158336 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5043113683968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83336724 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83336726 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108356700 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _runtime‚4511.703913268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27416694 trainer/global_step‚5749 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚333384388155392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191693424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329451692005512 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27417143 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4788980813524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76035570 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4788326375560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚338190404217344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191713490 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369161650204672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872Öfk 2Ò- _wandbruntime‚4511  _runtime‚4511—ZáÄYÓÓ %loss/total_loss‚46.23736270916264 _step‚471 2loss/SRqqt4/assignment_loss‚1.0920438463269155 /loss/FBt2/detection_loss‚2.4009935030570397 trainer/global_step‚5749 % _timestamp‚1.7884210119674788e+09 2loss/SRqqt3/assignment_loss‚1.3731080039507813 /loss/FRt4/detection_loss‚1.5983377921043849 1loss/SRqqt1/detection_loss‚2.3528140301887808 /loss/FBt1/detection_loss‚3.9133307576179504 .loss/FBt4/detection_loss‚0.699655649091663 .loss/FRt1/assignment_loss‚4.10064749514803 /loss/FBt3/detection_loss‚3.2910379648208616 epoch‚4 0loss/FBt2/assignment_loss‚0.2472709661636215 0loss/FBt1/assignment_loss‚0.4150583028792994 1loss/FBt3/assignment_loss‚0.35847201459109784 /loss/FRt1/detection_loss‚2.1642733152876508 1loss/SRqqt3/detection_loss‚1.9215821875466241 _runtime‚4511.704900391 0loss/FRt3/assignment_loss‚2.8368339002132417 .loss/FRt2/detection_loss‚2.798127053465162 0loss/SRqqt2/detection_loss‚2.599974725643794 1loss/SRqqt4/detection_loss‚0.7807869530176814 /loss/FRt2/assignment_loss‚4.035027664048331 0loss/FRt4/assignment_loss‚1.3661527164646836 2loss/SRqqt1/assignment_loss‚1.7293159656035595 2loss/SRqqt2/assignment_loss‚1.6839111057420573 0loss/FBt4/assignment_loss‚0.1308747031532463 .loss/FRt3/detection_loss‚2.347732093036175›M¾Ä^ÔØ 0loss/FBt1/assignment_loss‚0.4150583028792994 1loss/FBt3/assignment_loss‚0.35847201459109784 /loss/FRt1/detection_loss‚2.1642733152876508 1loss/SRqqt3/detection_loss‚1.9215821875466241 _runtime‚4511.704900391 0loss/FRt3/assignment_loss‚2.8368339002132417 .loss/FRt2/detection_loss‚2.798127053465162 0loss/SRqqt2/detection_loss‚2.599974725643794 1loss/SRqqt4/detection_loss‚0.7807869530176814 /loss/FRt2/assignment_loss‚4.035027664048331 0loss/FRt4/assignment_loss‚1.3661527164646836 2loss/SRqqt1/assignment_loss‚1.7293159656035595 2loss/SRqqt2/assignment_loss‚1.6839111057420573 0loss/FBt4/assignment_loss‚0.1308747031532463 .loss/FRt3/detection_loss‚2.347732093036175 %loss/total_loss‚46.23736270916264 _step‚471 2loss/SRqqt4/assignment_loss‚1.0920438463269155 /loss/FBt2/detection_loss‚2.4009935030570397 trainer/global_step‚5749 % _timestamp‚1.7884210119674788e+09 2loss/SRqqt3/assignment_loss‚1.3731080039507813 /loss/FRt4/detection_loss‚1.5983377921043849 1loss/SRqqt1/detection_loss‚2.3528140301887808 /loss/FBt1/detection_loss‚3.9133307576179504 .loss/FBt4/detection_loss‚0.699655649091663 .loss/FRt1/assignment_loss‚4.10064749514803 /loss/FBt3/detection_loss‚3.2910379648208616 epoch‚4 0loss/FBt2/assignment_loss‚0.2472709661636215סŠñ;Õ‚(:… —ÇäÔÈÛ«~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶Î5«zÖ‚(:ï —ÇäÔ…ã°€* disk./.usagePercent‚17.325820358734962 disk.rbd0.out‚0 disk.rbd1.out‚2.453125 disk.sda.out‚ 303.69140625! proc.memory.rssMB‚ 2771.390625% disk./.usageGB‚258.16724395751953 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ memory_percent‚93.57528686523438 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 network.recv‚ 1.71404e+06( disk.nvme0c0n1.out‚19025.80419921875 disk.sda2.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.915225982666016 cpu‚25.315619317950386 network.sent‚ 1.3026338e+07 disk.rbd0.in‚0 disk.sda3.out‚ 303.69140625 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚19025.80419921875$ proc.memory.availableMB‚1052.625 proc.cpu.threads‚17 disk.sda2.out‚0# disk.dm-0.out‚19025.80419921875 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 303.69140625 disk.drbd2266.in‚0n»Tj’ׂ(:‡ ¦ÇäÔŠŠ½~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.573' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ƒ‘koØ‚(:ä ¦Çäԇʤ! proc.memory.rssMB‚ 2771.390625 disk.sda.out‚ 304.3203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0$ memory_percent‚93.57681274414062% disk./.usageGB‚258.15274810791016 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 cpu‚25.314734371790326 disk.drbd1716.out‚0 disk.sda2.out‚0* disk./.usagePercent‚17.324847530104847 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 304.3203125 disk.dm-0.out‚19076.23828125* proc.memory.percent‚16.915225982666016 disk.rbd1.out‚2.453125$ proc.memory.availableMB‚1052.375 disk.sda1.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3134989e+07% disk.nvme0c0n1.out‚19076.23828125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 304.3203125 network.recv‚ 1.720535e+06# disk.nvme0n1.out‚19076.23828125 disk.dm-0.in‚ 4.33203125 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0×;f¯Ù‚(:’ µÇäÔ衬~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚187.374' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.474799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ý:mÚ‚(:â µÇäÔËù¡€ disk.sda.out‚304.75 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚304.75 network.recv‚ 1.724668e+06 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.57531070709229* proc.memory.percent‚16.915225982666016) disk./.usagePercent‚17.32673686570754% disk./.usageGB‚258.18090057373047# disk.dm-0.out‚19119.39794921875 disk.dm-1.in‚ 126.77734375 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2771.390625 disk.rbd1.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚19119.39794921875 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.sent‚ 1.3140863e+07 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1052.62109375& disk.nvme0n1.out‚19119.39794921875 cpu‚25.313877325145448 disk.dm-1.out‚304.75 disk.rbd1.out‚2.453125ëb6“2Û- _wandbruntime‚4547  _runtime‚4547’/ïqÉÜà trainer/global_step‚5799 % _timestamp‚1.7884210480971565e+09 _runtime‚4547.834709806 _step‚472 ' lr-AdamW/pg1‚0.00010484701981780619 ' lr-AdamW/pg2‚0.00010484701981780619¢#úâÎÝÈ _runtime‚4547.834709806 _step‚472 ' lr-AdamW/pg1‚0.00010484701981780619 ' lr-AdamW/pg2‚0.00010484701981780619 trainer/global_step‚5799 % _timestamp‚1.7884210480971565e+09ØwûL2Þ- _wandbruntime‚4547  _runtime‚4547}…hЀßÛO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340992543864320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336145687058944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367139594665472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5G>õªa#085431625728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 _runtime‚4547.835645281 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340992543864320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4828999717760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76660530 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884210480978885e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109284336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109284336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚5799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340991869498880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30õn˜èæ'ààO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340991869498880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4828345280004 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340992543864320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27641422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336145687058944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4846838802432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367139594665472 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5085431625728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4846182439936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84023508 _runtime‚4547.835645281 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109264298 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5085278487552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚340992543864320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327353685066752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚327353668027392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4846182439936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372224873153024 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332182684784512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4828999717760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109264298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340991869498880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336145687058944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76660530 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84023510 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884210480978885e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109284336 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49019108 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109284336 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49019110 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336145705061888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚5799 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336208384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚473 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193287806 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84023510 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372225050231296 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944ÙäXi2á- _wandbruntime‚4548  _runtime‚4548öô ¶ë&âåM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367201269343232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4829161714968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27646002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372287568341504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚474 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336202040097792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109282752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367201245403136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336202058100736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4829816152932 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109302816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49027276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚5799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 _runtime‚4548.504531557 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84037524 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884210487664583e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Íz­y ãêM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341049716344320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847001867776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193320276 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372287372402176 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109302816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84037526 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84037524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109282752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327408589197312 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336202058100736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5086298998272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76673739 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109282752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193340342 E?DeviceStatsMonitor.on_train_batch_end/oversize_aXKNwllocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367201245403136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193320276 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327408572157952 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336202058100736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4829816152932 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84037526 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5086126999040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847658243584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49027278 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109302816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49027276 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚5799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332238405350244 _runtime‚4548.504531557 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193340342 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84037524 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884210487664583e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4847001867776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367201269343232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4829161714968 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76673278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27646002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372287568341504 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚474 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336202040097792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330098874555392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110190350 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338963357004288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110210388 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4869835057168 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84724308 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334968726651920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343851181612544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128433687040 _step‚477 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49427510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338963357004288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343850507247104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334968726651920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343851181612544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5128433687040 _step‚477 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343850507247104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49427510 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338963357004288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330098874555392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110190350 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338963357004288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887824608256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110210388 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4869835057168 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84724308 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110190350 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375350565235712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4869180619412 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84738326 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339019710043136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27874882 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330153795725312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884210850731127e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84738324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚478 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110208804 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110228868 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339019692040192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27875334 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375413065640960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84738324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84738326 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110228868 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521Ý[¾†ð&óêM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚478 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888644049408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887987673600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194947128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110208804 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110228868 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339019692040192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110208804 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129451159552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335023775740328 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27875334 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚370283782286336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4869997054376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375413065640960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚370283806226432 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84738324 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336208384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4888644049408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339019692040192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343908354092544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84738326 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110228868 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚4584.811102823 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4887987673600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77310558 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5129283354624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194967194 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84738326 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚330153778685952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339019710043136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194967194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27874882 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚330153795725312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343907679713792 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884210850731127e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84738324 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49435678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375413257385984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680Þ.«o2ô- _wandbruntime‚4584  _runtime‚4584 ‰»b\õÖ .loss/FRt3/detection_loss‚2.149682474732399 1loss/FBt3/assignment_loss‚0.05965263534744736 2loss/SRqqt3/assignment_loss‚1.5887226651197877 /loss/FRt1/detection_loss‚2.6286296868646466 /loss/FRt4/detection_loss‚1.8928778622156153 0loss/SRqqt4/detection_loss‚1.135994440802033 .loss/FBt3/detection_loss‚2.367947429418564 _runtime‚ 4584.81258024 /loss/FRt2/assignment_loss‚3.931172503874852 0loss/FRt3/assignment_loss‚2.7023635792732237 1loss/SRqqt1/detection_loss‚2.4670601554455294 1loss/SRqqt3/detection_loss‚1.9795279623619442 _step‚479 1loss/FBt1/assignment_loss‚0.16330233520511683 1loss/FBt2/assignment_loss‚0.37180688627618963 &loss/total_loss‚46.486841061386805 epoch‚4 % _timestamp‚1.7884210850748491e+09 2loss/SRqqt4/assignment_loss‚0.9466456464151778 0loss/SRqqt2/detection_loss‚2.363913454792716 trainer/global_step‚5849 0loss/FRt4/assignment_loss‚1.8296894350877175 2loss/SRqqt2/assignment_loss‚2.1152697731599663 1loss/FBt4/assignment_loss‚0.09871790988425355 /loss/FRt2/detection_loss‚3.0450976536824155 /loss/FBt1/detection_loss‚2.9633470873037973 /loss/FBt2/detection_loss‚2.8592163026332855 .loss/FBt4/detection_loss‚0.665425004542894 /loss/FRt1/assignment_loss‚4.542853073493855 2loss/SRqqt1/assignment_loss‚1.6179251034533786\òtkaöÛ 1loss/FBt2/assignment_loss‚0.37180688627618963 &loss/total_loss‚46.486841061386805 epoch‚4 % _timestamp‚1.7884210850748491e+09 2loss/SRqqt4/assignment_loss‚0.9466456464151778 0loss/SRqqt2/detection_loss‚2.363913454792716 trainer/global_step‚5849 0loss/FRt4/assignment_loss‚1.8296894350877175 2loss/SRqqt2/assignment_loss‚2.1152697731599663 1loss/FBt4/assignment_loss‚0.09871790988425355 /loss/FRt2/detection_loss‚3.0450976536824155 /loss/FBt1/detection_loss‚2.9633470873037973 /loss/FBt2/detection_loss‚2.8592163026332855 .loss/FBt4/detection_loss‚0.665425004542894 /loss/FRt1/assignment_loss‚4.542853073493855 2loss/SRqqt1/assignment_loss‚1.6179251034533786 .loss/FRt3/detection_loss‚2.149682474732399 1loss/FBt3/assignment_loss‚0.05965263534744736 2loss/SRqqt3/assignment_loss‚1.5887226651197877 /loss/FRt1/detection_loss‚2.6286296868646466 /loss/FRt4/detection_loss‚1.8928778622156153 0loss/SRqqt4/detection_loss‚1.135994440802033 .loss/FBt3/detection_loss‚2.367947429418564 _runtime‚ 4584.81258024 /loss/FRt2/assignment_loss‚3.931172503874852 0loss/FRt3/assignment_loss‚2.7023635792732237 1loss/SRqqt1/detection_loss‚2.4670601554455294 1loss/SRqqt3/detection_loss‚1.9795279623619442 _step‚479 1loss/FBt1/assignment_loss‚0.16330233520511683ß zfH’÷‚(:‡ âÇäÔúÚ«~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n#'tø‚(:é âÇäÔÍ‹Ó& disk.nvme0c0n1.out‚19264.154296875 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 306.3046875 disk.dm-1.out‚ 306.3046875 disk.sda.in‚ 126.77734375$ disk.nvme0n1.out‚19264.154296875 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚93.57829093933105 cpu‚25.311289055582876 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2266.out‚0 disk.sda.out‚ 306.3046875 disk.nvme0n1.in‚ 4.33203125( proc.memory.availableMB‚ 1052.1328125 network.sent‚ 1.3261027e+07 network.recv‚ 1.73918e+06 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚17.32967583220062% disk./.usageGB‚258.22469329833984 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda1.out‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2265.in‚0! disk.dm-0.out‚19264.154296875 disk.drbd1716.in‚0ºN6uù‚(:’ ñÇäÔÅ® ~ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;a§uú‚(:ê ñÇäÔÜû“€ disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 306.7890625 disk.drbd2266.out‚0 network.recv‚ 1.745187e+06) disk./.usagePercent‚17.33052398516156 disk.rbd1.in‚0 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚19305.888671875) proc.memory.availableMB‚ 1052.12890625 disk.sda1.out‚0 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.3369542e+07 disk.sda2.in‚0 disk.drbd2080.in‚0! disk.dm-0.out‚19305.888671875* proc.memory.percent‚16.915225982666016 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 306.7890625 proc.cpu.threads‚17 disk.rbd1.out‚2.453125 disk.nvme0c0n1.in‚ 4.33203125$ memory_percent‚93.57831478118896% disk./.usageGB‚258.23733139038086 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.390625 disk.rbd0.out‚0 disk.drbd2265.out‚0 cpu‚25.31045991880586 disk.sda.out‚ 306.7890625$ disk.nvme0n1.out‚19305.888671875 disk.rbd0.in‚0k\i%’û‚(:‡ €ÈäÔ癫~ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3398 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Kë5yü‚(:î €ÈäÔ–Ì°€ network.sent‚ 1.3375843e+07 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# disk.dm-0.out‚19360.84228515625 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 4.33203125 cpu‚25.30950879613077 network.recv‚ 1.749943e+06* disk./.usagePercent‚17.332366215377952 disk.dm-1.out‚ 307.61328125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚19360.84228515625 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 307.61328125$ proc.memory.availableMB‚1051.875 disk.drbd2080.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚93.57986450195312 proc.cpu.threads‚17 disk.sda.out‚ 307.61328125 disk.sda2.out‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016$ disk./.usageGB‚258.2647819519043 disk.dm-0.in‚ 4.33203125 disk.sda1.out‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚19360.84228515625Ñ°s2ý- _wandbruntime‚4620  _runtime‚4620ŸÄ1Éþà _runtime‚4620.380143502 _step‚480 ' lr-AdamW/pg1‚0.00010247667065910816 ' lr-AdamW/pg2‚0.00010247667065910816 trainer/global_step‚5899 % _timestamp‚1.7884211206425922e+09L*ÉÎÿÈ % _timestamp‚1.7884211206425922e+09 _runtime‚4620.380143502 _step‚480 ' lr-AdamW/pg1‚0.00010247667065910816 ' lr-AdamW/pg2‚0.00010247667065910816 trainer/global_step‚5899àÆh0M2€- _wandbruntime‚4620  _runtime‚4620ÄËNÔÛO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373304668431872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4910670396576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85425110 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111136440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4620.381077424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4928154051584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85425110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337754768519328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884211206433113e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373304692371968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440O&l  UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77935779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85425108 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49835910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378476583461888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332844081083392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341780990943744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28099414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414³EZæ'‚àO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4928810414080 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77935322 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85425108 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111136440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346709144995328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5171891089920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4928810414080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111116402 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77935779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85425108 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5171725368832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49835910 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378476583461888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341781008946688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28099869 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332844081083392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341780990943744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28099414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196541510 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚332844098122752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4910015958820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346709819360768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373304668431872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4910670396576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85425110 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4928154051584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111136440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4620.381077424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341780990943744 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚346709819360768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4928154051584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85425110 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337754768519328 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111116402 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5899 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49835908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884211206433113e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346709144995328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373304692371968ábÉ2ƒ- _wandbruntime‚4621  _runtime‚4621ü¿2Ûë&„åM trainer/global_step‚5899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884211213183687e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4621.056519044 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196573980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337809817607736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414›W)Í…êM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196573980 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85439126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28103901 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337809817607736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111134856 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4928973479424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332898985213952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378538903299072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341837343982592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378539095044096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚5899 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85439124 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346766991840768 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49844076 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111154920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5172751934464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884211213183687e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _step‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341837361985536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4929629855232 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373366343109632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚4621.056519044 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4929629855232 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77947977 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337810489085060 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346766317462016 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196594046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196573980 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49844078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsM¬ñD[#onitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196594046 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111154920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346766991840768 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77948452 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111134856 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430âþbÊ2†- _wandbruntime‚4621  _runtime‚4621?É\‡Ö /loss/FRt3/detection_loss‚2.1897537133523395 trainer/global_step‚5899 _step‚483 .loss/FBt1/detection_loss‚3.521125543117523 2loss/SRqqt3/assignment_loss‚1.7380588611510273 1loss/FBt2/assignment_loss‚0.07927801109898017 /loss/FBt4/detection_loss‚0.5428191431940497 epoch‚4 0loss/FRt1/assignment_loss‚3.0474767608845488 /loss/FRt2/assignment_loss‚3.572463265780745 2loss/SRqqt1/assignment_loss‚1.9540424587783114 0loss/FRt4/assignment_loss‚1.5354835542978025 1loss/FBt4/assignment_loss‚0.16598345476321227 1loss/SRqqt2/detection_loss‚2.4366010273656538 1loss/SRqqt3/detection_loss‚1.7693347601840894 /loss/FBt3/detection_loss‚3.3093192100524904 % _timestamp‚1.7884211213201377e+09 .loss/FRt2/detection_loss‚2.929447459763494 1loss/SRqqt2/assignment_loss‚1.756548963246807 /loss/FRt4/detection_loss‚1.8201836924634727 1loss/SRqqt1/detection_loss‚1.8248121296537334 1loss/SRqqt4/detection_loss‚1.0639639158987186 .loss/FBt2/detection_loss‚2.802510972533907 %loss/total_loss‚43.94837700296909 1loss/FBt3/assignment_loss‚0.17281019217252833 _runtime‚4621.057896313 0loss/FRt3/assignment_loss‚2.4548300419534956 1loss/SRqqt4/assignment_loss‚0.915886702231188 0loss/FBt1/assignment_loss‚0.1353542195616683 /loss/FRt1/detection_loss‚2.2102889494693025't¢-aˆÛ 2loss/SRqqt1/assignment_loss‚1.9540424587783114 0loss/FRt4/assignment_loss‚1.5354835542978025 1loss/FBt4/assignment_loss‚0.16598345476321227 1loss/SRqqt2/detection_loss‚2.4366010273656538 1loss/SRqqt3/detection_loss‚1.7693347601840894 /loss/FBt3/detection_loss‚3.3093192100524904 % _timestamp‚1.7884211213201377e+09 .loss/FRt2/detection_loss‚2.929447459763494 1loss/SRqqt2/assignment_loss‚1.756548963246807 /loss/FRt4/detection_loss‚1.8201836924634727 1loss/SRqqt1/detection_loss‚1.8248121296537334 1loss/SRqqt4/detection_loss‚1.0639639158987186 .loss/FBt2/detection_loss‚2.802510972533907 %loss/total_loss‚43.94837700296909 1loss/FBt3/assignment_loss‚0.17281019217252833 _runtime‚4621.057896313 0loss/FRt3/assignment_loss‚2.4548300419534956 1loss/SRqqt4/assignment_loss‚0.915886702231188 0loss/FBt1/assignment_loss‚0.1353542195616683 /loss/FRt1/detection_loss‚2.2102889494693025 /loss/FRt3/detection_loss‚2.1897537133523395 trainer/global_step‚5899 _step‚483 .loss/FBt1/detection_loss‚3.521125543117523 2loss/SRqqt3/assignment_loss‚1.7380588611510273 1loss/FBt2/assignment_loss‚0.07927801109898017 /loss/FBt4/detection_loss‚0.5428191431940497 epoch‚4 0loss/FRt1/assignment_loss‚3.0474767608845488 /loss/FRt2/assignment_loss‚3.572463265780745ãñÄK’‰‚(:‡ ÈäÔ÷Ö¡~ gpu.0.gpu‚56.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.794' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7588 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©v˜ø|Š‚(:ñ ÈäÔ¤žˆ# disk.dm-0.out‚19413.70166015625 disk.drbd2266.out‚0 disk.rbd1.out‚2.453125 disk.drbd1716.in‚0$ memory_percent‚93.57879161834717! proc.memory.rssMB‚ 2771.390625 disk.sda2.out‚0 disk.sda3.out‚ 308.20703125 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 308.20703125 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚19413.70166015625 network.recv‚ 1.75318e+06 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.333746095987507 disk.dm-0.in‚ 4.33203125 disk.drbd1716.out‚0 network.sent‚ 1.3381178e+07 disk.rbd0.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.915225982666016 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 4.33203125( disk.nvme0c0n1.out‚19413.70166015625 disk.sda.in‚ 126.77734375) proc.memory.availableMB‚ 1052.05078125# disk./.usageGB‚258.285343170166 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 308.20703125 disk.nvme0n1.in‚ 4.33203125 cpu‚25.30866603533683W&’‹‚(:‡ žÈäÔ‹·‰~ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ -{Œ‚(:ð žÈäÔìÕþ€ disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375! proc.memory.rssMB‚ 2771.390625 disk.drbd1716.out‚0& disk.nvme0n1.out‚19462.04638671875 disk.drbd2080.in‚0 cpu‚25.30778369915866) disk./.usagePercent‚17.33473710115255( disk.nvme0c0n1.out‚19462.04638671875 disk.drbd2080.out‚0% disk./.usageGB‚258.30010986328125 disk.sda1.in‚0# disk.dm-0.out‚19462.04638671875 disk.drbd2266.out‚0 disk.sda.out‚ 308.8828125 network.sent‚ 1.3489759e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚2.453125 disk.nvme0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 308.8828125 disk.rbd1.in‚0$ memory_percent‚93.57879161834717 network.recv‚ 1.759738e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 308.8828125 disk.sda.in‚ 126.77734375) proc.memory.availableMB‚ 1052.05078125* proc.memory.percent‚16.915225982666016 disk.nvme0c0n1.in‚ 4.33203125;´šx2- _wandbruntime‚4656  _runtime‚46569‹àDÉŽÃ _runtime‚4656.462340652 _step‚484 ' lr-AdamW/pg1‚0.00010128570392584378 ' lr-AdamW/pg2‚0.00010128570392584378 trainer/global_step‚5949 % _timestamp‚1.7884211567247953e+09ÊêAÁÎÈ _runtime‚4656.462340652 _step‚484 ' lr-AdamW/pg1‚0.00010128570392584378 ' lr-AdamW/pg2‚0.00010128570392584378 trainer/global_step‚5949 % _timestamp‚1.7884211567247953e+09äÄœiž2- _wandbruntime‚4656  _runtime‚46564*m}à'‘ÚO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50244310 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78573286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5214798180352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344598660889088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28329427 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112062492 $ _timestamp‚1.788421156725497e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376387229255168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86125910 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381602184768000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5214955512832 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340540810386736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4951505735984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349568457108992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚5949 _runtime‚4656.463222737 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112062492 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50244308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28328978 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335589304650752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86125910 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚485 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376387205315072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408¶õÛßå'’ßO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4969796219904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376387205315072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349567782743552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4969139857408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50244310 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198188402 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78573286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5214798180352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112042454 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚344598660889088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28329427 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4969796219904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112062492 $ _timestamp‚1.788421156725497e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376387229255168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚335589287611392 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86125910 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86125908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚381602184768000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5214955512832 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340540810386736 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4951505735984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349568457108992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚5949 _runtime‚4656.463222737 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112062492 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349568457108992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50244308 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28328978 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚335589304650752 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86125910 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚485 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038å`àÆ2“- _wandbruntime‚4657  _runtime‚4657S/Ù!q”åM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198220898 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _runtime‚4657.362403334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376448879992832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78586453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚486 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335644191741952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344655013927936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112060908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349624955210240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215815272960ÕNïð&•êM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376448856052736 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4970615661056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381664695265792 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112080972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28333514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198220898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198220898 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚344655013927936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4969959285248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50252476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 _runtime‚4657.362403334 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚463 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376448879992832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78586453 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚340596530952468 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _step‚486 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198200832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78585990 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381664497229312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚335644191741952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349625629588992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚344655013927936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4970615661056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50252478 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112080972 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112060908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349624955210240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5215815272960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4969959285248 F2 - _wandbruntime‚4692  _runtime‚4692cw‘j,!¡ÚO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343326180777028 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4692.708612877 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352427094857216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50652710 ?9DeviceStatsMonitor.on_train_batch_start/num_a? ,Ø´lloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5999 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025õ¿ãùå'¢ßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚437 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5257935678464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343326180777028 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4692.708612877 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚347416294828544 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚379469742198272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚379469766138368 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199795214 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384727701816832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384727518447104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚352426420491776 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347416312831488 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343326852254144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5010125663232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86826710 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112988544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199815254 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79210060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚352427094857216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚352427094857216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50652710 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112988544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚5999 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚489 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86826708 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28557352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338334511178752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86826708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112968506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338334494139392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50652708 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28557787 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116é'ïf2£- _wandbruntime‚4693  _runtime‚4693ñ=íÊë&¤åM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199847750 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011601466880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338389415309312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976lÊîð&¥êM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86840724 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79223075 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199827684 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5011601466880 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚347472665870336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338389415309312 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113007024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79222647 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112986960 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5258659774976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384790052710912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5258806608384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199847750 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚379531392935936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚379531416876032 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4992503072600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86840726 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343382572819876 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199847750ê»v´¹2¦- _wandbruntime‚4693  _runtime‚4693ÍpœÔ÷§× 2loss/SRqqt1/assignment_loss‚1.8243316471576692 2loss/SRqqt2/assignment_loss‚2.6958066040071946 0loss/FBt1/assignment_loss‚0.4457749602281385 1loss/FBt4/assignment_loss‚0.12236137543258756 /loss/FRt2/detection_loss‚2.7769780255652763 0loss/SRqqt4/detection_loss‚1.048497781265473 trainer/global_step‚5999 0loss/FRt2/assignment_loss‚3.9337973320806348 1loss/SRqqt2/detection_loss‚2.3790941906386407 1loss/SRqqt3/assignment_loss‚1.353883421211912 /loss/FBt2/detection_loss‚3.0948198214173317 /loss/FRt1/assignment_loss‚2.651770930541189 .loss/FBt3/detection_loss‚2.066695417676653 /loss/FRt1/detection_loss‚2.5643933054647947 /loss/FRt3/detection_loss‚2.3313352194097305 1loss/SRqqt3/detectiV…äfon_loss‚1.6468541373809178 &loss/total_loss‚44.284350743016766 0loss/FRt4/assignment_loss‚1.6413349851648857 /loss/FBt1/detection_loss‚2.4983399179246693 epoch‚4 0loss/FRt3/assignment_loss‚2.6007234613100687 1loss/SRqqt4/assignment_loss‚1.210842171009036 1loss/FBt2/assignment_loss‚0.35381391271948814 _runtime‚4693.393117271 _step‚491 1loss/FBt3/assignment_loss‚0.45725299544366343 .loss/FRt4/detection_loss‚1.944147377889207 1loss/SRqqt1/detection_loss‚2.0733436230156155 /loss/FBt4/detection_loss‚0.5681581290619862 % _timestamp‚1.7884211936556702e+09ÿKb¨Ü 0loss/FRt3/assignment_loss‚2.6007234613100687 1loss/SRqqt4/assignment_loss‚1.210842171009036 1loss/FBt2/assignment_loss‚0.35381391271948814 _runtime‚4693.393117271 _step‚491 1loss/FBt3/assignment_loss‚0.45725299544366343 .loss/FRt4/detection_loss‚1.944147377889207 1loss/SRqqt1/detection_loss‚2.0733436230156155 /loss/FBt4/detection_loss‚0.5681581290619862 % _timestamp‚1.7884211936556702e+09 2loss/SRqqt1/assignment_loss‚1.8243316471576692 2loss/SRqqt2/assignment_loss‚2.6958066040071946 0loss/FBt1/assignment_loss‚0.4457749602281385 1loss/FBt4/assignment_loss‚0.12236137543258756 /loss/FRt2/detection_loss‚2.7769780255652763 0loss/SRqqt4/detection_loss‚1.048497781265473 trainer/global_step‚5999 0loss/FRt2/assignment_loss‚3.9337973320806348 1loss/SRqqt2/detection_loss‚2.3790941906386407 1loss/SRqqt3/assignment_loss‚1.353883421211912 /loss/FBt2/detection_loss‚3.0948198214173317 /loss/FRt1/assignment_loss‚2.651770930541189 .loss/FBt3/detection_loss‚2.066695417676653 /loss/FRt1/detection_loss‚2.5643933054647947 /loss/FRt3/detection_loss‚2.3313352194097305 1loss/SRqqt3/detection_loss‚1.6468541373809178 &loss/total_loss‚44.284350743016766 0loss/FRt4/assignment_loss‚1.6413349851648857 /loss/FBt1/detection_loss‚2.4983399179246693 epoch‚4ëè2œ©‚(:‘ ËÈäÔ‡™~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.372' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.27440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“6§uª‚(:ê ËÈäÔ‰ž«€ disk.dm-1.out‚ 310.734375 network.recv‚ 1.775176e+06 disk.sda1.out‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚19626.5888671875 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.out‚ 310.734375" disk.dm-0.out‚19626.5888671875 disk.drbd2266.out‚0* proc.memory.percent‚16.915225982666016% disk./.usageGB‚258.33190536499023 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0 disk.rbd1.in‚0 network.sent‚ 1.3610118e+07 disk.rbd0.in‚0 disk.sda2.in‚0 cpu‚25.305280798485644* disk./.usagePercent‚17.336870923950443 disk.sda.out‚ 310.734375 disk.sda1.in‚0' disk.nvme0c0n1.out‚19626.5888671875 disk.drbd2080.in‚0 disk.drbd2080.out‚0' proc.memory.availableMB‚ 1051.703125 disk.nvme0n1.in‚ 4.33203125 disk.dm-0.in‚ 4.33203125$ memory_percent‚93.58091354370117 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2771.390625 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.rbd1.out‚2.453125ó:!_’«‚(:‡ ÚÈäԪњ~ gpu.0.gpu‚53.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.379' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.2758 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i¾HÕt¬‚(:é ÚÈäÔ·ŠŽ disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0% disk.nvme0c0n1.out‚19691.03515625 disk.sda3.in‚ 126.77734375$ memory_percent‚93.57788562774658 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda.out‚ 311.23046875 network.sent‚ 1.3718711e+07 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 1.781462e+06 disk.nvme0n1.in‚ 4.33203125# disk.nvme0n1.out‚19691.03515625 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 311.23046875) proc.memory.availableMB‚ 1052.19921875! proc.memory.rssMB‚ 2771.390625 disk.dm-0.in‚ 4.33203125 disk.dm-1.out‚ 311.23046875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚2.453125 disk.sda1.out‚0) disk./.usagePercent‚17.33976022498188$ disk./.usageGB‚258.3749580383301 disk.drbd2265.in‚0 disk.dm-0.out‚19691.03515625 disk.dm-1.in‚ 126.77734375 disk.sda2.out‚0* proc.memory.percent‚16.915225982666016 disk.drbd1716.out‚0 disk.sda2.in‚0 cpu‚25.30441099392341˜¼a°’­‚(:‡ éÈäÔíÚ´~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yBÿÍl®‚(:á éÈäÔ…Ê€) proc.memory.availableMB‚ 1051.70703125 disk.sda1.out‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚93.58088970184326 disk.dm-1.out‚ 311.77734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda3.out‚ 311.77734375 disk.dm-0.out‚ 19737.59375 network.recv‚ 1.785649e+06 disk.nvme0n1.out‚ 19737.59375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 cpu‚25.30361362186612* disk./.usagePercent‚17.335193050571032$ disk./.usageGB‚258.3069038391113 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda.out‚ 311.77734375 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016 network.sent‚ 1.3724819e+07 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 19737.59375 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0éHÝœ¯‚(:… øÈäÔ½àÿ} gpu.0.gpu‚53.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^hô’€°‚(:õ øÈäÔ“ˆ¡€% disk./.usageGB‚258.42087173461914 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.318135427046137 network.recv‚ 1.790162e+06 disk.drbd1716.in‚0 disk.dm-1.in‚ 126.77734375! proc.memory.rssMB‚ 2771.390625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚19790.38720703125 disk.drbd2080.out‚0 disk.dm-1.out‚ 312.30078125 disk.drbd2266.out‚0 disk.rbd1.out‚2.453125$ memory_percent‚93.58394145965576* disk./.usagePercent‚17.342841531663993 disk.rbd0.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1051.20703125 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 312.30078125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 312.30078125 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 network.sent‚ 1.3731128e+07 disk.nvme0n1.in‚ 4.33203125& disk.nvme0n1.out‚19790.38720703125# disk.dm-0.out‚19790.38720703125 disk.drbd2266.in‚0* proc.memory.percent‚16.915225982666016¸×#±‚(:’ ‡ÉäÔ‚º~ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚188.731' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.746199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ?_z²‚(:ï ‡ÉäÔÇâé€ disk.sda1.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.nvme0c0n1.in‚ 4.33203125 network.recv‚ 1.792738e+06$ disk./.usageGB‚258.4372253417969 disk.drbd1716.out‚0$ memory_percent‚93.57943534851074 cpu‚25.331667371071617 proc.cpu.threads‚17 disk.drbd2080.out‚0# disk.dm-0.out‚19846.93798828125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2771.390625 disk.drbd2265.out‚0 disk.dm-1.out‚ 312.9609375 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0& disk.nvme0n1.out‚19846.93798828125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚19846.93798828125) disk./.usagePercent‚17.34393903596328 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 1051.9453125* proc.memory.percent‚16.915225982666016 disk.drbd2080.in‚0 disk.sda.out‚ 312.9609375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.3735486e+07 disk.sda3.out‚ 312.9609375 disk.dm-0.in‚ 4.33203125 disk.rbd1.out‚2.453125¸P° ‘³‚(:† –ÉäÔèó¤~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0] yl´‚(:á –ÉäÔî¹´ disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2080.out‚0$ memory_percent‚93.57943534851074 disk.rbd0.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 1051.9453125$ disk.nvme0n1.out‚19903.869140625 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2265.in‚0 cpu‚25.346365543993375 disk.drbd2080.in‚0! disk.dm-0.out‚19903.869140625 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚2.453125 network.sent‚ 1.3741617e+07 disk.sda.in‚ 126.77734375 disk.sda1.out‚0& disk.nvme0c0n1.out‚19903.869140625 disk.sda3.out‚313.3125 disk.dm-1.out‚313.3125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.dm-0.in‚ 4.33203125 disk.sda.out‚313.3125 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0! proc.memory.rssMB‚ 2771.390625* proc.memory.percent‚16.915225982666016 proc.cpu.threads‚17 network.recv‚ 1.797093e+06* disk./.usagePercent‚17.344689650053677$ disk./.usageGB‚258.4484100341797 disk.rbd0.out‚0«ùT"2µ- _wandbruntime‚4772  _runtime‚4772)êåeH*¶ÂT QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 _step‚492 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚80338713 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚347636727186428 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5047739740756 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚342589495399424 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5047248826364 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚384161611012608 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚88000940 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚351813224029696 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 trainer/global_step‚6044 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 114196905 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚351813224029696 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 202181866 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 119538176 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202181866 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚88000940 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 114180928 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚342589478360064 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚356879105094144 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202197845 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 114180928 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5066391897600 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5315574679552 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚389477185692160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143478272 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚356879105094144 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚351813206026752 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5066391897600 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚356879615927296 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚161 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 202197845 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51347132 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚28991740 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚389477185692160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 143478272 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚88000938 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚356879105094144 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚351813206026752 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5066391897600 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚356879615927296 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚161 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 202197845 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚51347132 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚28991740 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 114204207 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 _step‚493 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 114204207 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚389515426552320 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5066264194048 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚88011746 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚88011744 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚51353636 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 5315939806208 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚342624772119552 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚80349799 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚384199486746112 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚347672384555324 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 114188230 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 % _timestamp‚1.7884212741492834e+09 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 114188230 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5315820268032 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚88011746 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 5066757024256 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚28996163 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚347672892509076 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚356916300249600 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202215953 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 114204207 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 202199974 _step‚493 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 114204207 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 202215953 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚389515426552320 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚28996322 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430í'ѳb’»‚(:‡ ¥ÉäÔ껪~ gpu.0.gpu‚50.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚152.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µï’rt¼‚(:é ¥ÉäÔ¥‘¿€ disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 313.94921875$ disk./.usageGB‚258.4582748413086 disk.drbd1716.in‚0$ disk.nvme0n1.out‚19953.115234375 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 1.3747502e+07& disk.nvme0c0n1.out‚19953.115234375 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 313.94921875! disk.dm-0.out‚19953.115234375! proc.memory.rssMB‚ 2771.390625 cpu‚25.360728501171483 proc.cpu.threads‚17 disk.drbd2080.out‚0 network.recv‚ 1.801146e+06 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2266.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0& proc.memory.availableMB‚ 1051.65625* proc.memory.percent‚16.915225982666016 disk.rbd1.out‚2.453125 disk.sda.out‚ 313.94921875 disk.dm-0.in‚ 4.33203125# memory_percent‚93.5811996459961) disk./.usagePercent‚17.34535168553722(¾2½- _wandbruntime‚4786  _runtime‚4786È¢âv%¾÷§ ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_5‚0.02150537632405758 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2737075984477997 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5453597903251648 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3444959223270416 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17456413805484772 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.965732216835022 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259581565857 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5470290184020996 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9854181408882141 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39044734835624695 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5425363779067993 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6042498350143433 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17263108491897583 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994246780872345 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5479544401168823 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20203690230846405 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.134332537651062 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11018085479736328 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014016074128448963 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5311877727508545 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10660479217767715 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6394432187080383 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5345146656036377 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0956488624215126 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5640880465507507 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876686930656433 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6403805613517761 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3701569736003876 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03905374929308891 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.622412383556366 +particle/sensitivity‚0.2762800455093384 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.586309552192688 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5721470713615417 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9824585318565369 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08160187304019928 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0046040378510952 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.48725494742393494 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.977398157119751 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020674937404692173 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24197319149971008 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5151730179786682 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5892857313156128 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2806496322154999 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1158113181591034 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5948275923728943 *jet/accuracy_2_of_5‚0.4623016119003296 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.97284996509552 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11083588004112244 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9828663468360901 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3649912178516388 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45955169200897217 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35505053400993347 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5847222805023193 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9533045291900635 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1029883623123169 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5435823798179626 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19750964641571045 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9306416511535645 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4618476629257202 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3474701941013336 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08024691045284271 ("Purity/4FRt1SRqqt0FBt/SRqqt_purph1ˆnity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.194403275847435 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15202702581882477 particle/accuracy_1_of_2‚1 % _timestamp‚1.7884212868607614e+09 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01972789317369461 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5580557584762573 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 _runtime‚ 4786.60049313 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6097093820571899 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5932203531265259 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8878276944160461 particle/accuracy_4_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9752041101455688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9896442890167236 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5460326671600342 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.973099410533905 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 jet/accuracy_6_of_7‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2520114779472351 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9857735633850098 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9847959876060486 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5659120082855225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13887634873390198 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980158805847168 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15060457587242126 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37931621074676514 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16570083796977997 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5390766263008118 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.521402895450592 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5083557963371277 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.525708019733429 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915282726287842 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9785513877868652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9875161051750183 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.46666663885116577 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +jet/accuracy_3_of_6‚0.17307692766189575 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.968148410320282 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18691521883010864 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 jet/accuracy_2_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08050190657377243 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868852496147156 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3121297359466553 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4783046245574951 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984534382820129 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_6_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.575225830078125 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5407142639160156 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_7‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.824967622756958 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.49629637598991394 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14217491447925568 particle/accuracy_3_of_3‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5655006766319275 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5694010257720947 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956281185150146 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.207628071308136 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9985570311546326 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03839286044239998 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1371840536594391 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996625542640686 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 *jet/accuracy_1_of_6‚0.9807692170143127 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16503910720348358 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35520830750465393 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 epoch‚4 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5692957043647766 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5786595940589905 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6590000987052917 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5083380341529846 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14141415059566498 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02051088772714138 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47054973244667053 *jet/accuracy_2_of_4‚0.3506696820259094 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 particle/accuracy_4_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25574010610580444 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011835474520921707 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5760112404823303 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5661501288414001 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6234567165374756 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18876373767852783 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11817921698093414 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941776990890503 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2936480939388275 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5692957043647766 +particle/specificity‚0.9409050941467285 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32315659523010254 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3720930218696594 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6574074625968933 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00692239822819829 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5260360240936279 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07136532664299011 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654887199402 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010754982009530067 )jet/accuracy_1_of_1‚0.512367844581604 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.024256346747279167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5874642729759216 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 particle/accuracy_5_of_5‚1 (particle/accuracy‚0.8371208310127258 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5274621844291687 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1383114904165268 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9597734212875366 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11783625930547714 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02058052457869053 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2331349104642868 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16120398044586182 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12814311683177948 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14262425899505615 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5215846300125122 +jet/accuracy_2_of_2‚0.10325626283884048 ,jet/accuracy_4_of_4‚0.006026997230947018 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5729286074638367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09502194076776505 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185670852661 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823567271232605 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12151843309402466 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.23586654663085938 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11347106099128723 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14543646574020386 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1587904840707779 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 trainer/global_step‚6044 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802686214447 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚494 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1587904840707779 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5842559337615967 particle/accuracy_1_of_1‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.4495014250278473 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4721282422542572 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 particle/accuracy_1_of_7‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008367364294826984 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚45.46630486502299 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2384743094444275 +jet/accuracy_2_of_6‚0.47435900568962097 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05619897320866585 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18364156782627106 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.026900924742221832 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42977678775787354 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5871469974517822 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47986429929733276 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5313296914100647 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4616858661174774 ,jet/accuracy_5_of_5‚0.008064515888690948 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5442843437194824 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48359137773513794 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5918498635292053 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13463521003723145 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45075076818466187 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6333333253860474 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24645274877548218 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09794341772794724 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 'particle/f_score‚0.3461245894432068 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 particle/accuracy_3_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45751872658729553 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38424691557884216 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15028735995292664 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3892543911933899 *jet/accuracy_1_of_2‚0.7730676531791687 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.046738673001527786 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_3_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5603658556938171 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3497682511806488 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2977462410926819 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20455457270145416 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24792563915252686 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14511340856552124 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2096046507358551 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 particle/accuracy_4_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3083333671092987 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 particle/accuracy_5_of_6‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3778842091560364Å Ú§j¿ü§ =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01972789317369461 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5580557584762573 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 _runtime‚ 4786.60049313 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.6097093820571899 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5932203531265259 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8878276944160461 particle/accuracy_4_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9752041101455688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9896442890167236 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5460326671600342 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.973099410533905 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 jet/accuracy_6_of_7‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2520114779472351 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 * Purity/0FRt2SRqqt2FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9857735633850098 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9847959876060486 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5659120082855225 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13887634873390198 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9980158805847168 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15060457587242126 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37931621074676514 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16570083796977997 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5390766263008118 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.521402895450592 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.016352200880646706 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5083557963371277 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.525708019733429 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915282726287842 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9785513877868652 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9875161051750183 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.46666663885116577 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +jet/accuracy_3_of_6‚0.17307692766189575 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.968148410320282 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18691521883010864 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Puriòƒäùty/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9233333468437195 jet/accuracy_2_of_7‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08050190657377243 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9868852496147156 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3121297359466553 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.0625 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4783046245574951 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3854166865348816 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984534382820129 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_6_of_7‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.575225830078125 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5407142639160156 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_7‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.824967622756958 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.49629637598991394 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14217491447925568 particle/accuracy_3_of_3‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5655006766319275 <"Purity/2FRt*SRqqt2FBt/event_purity‚0.0058139534667134285 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5694010257720947 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9956281185150146 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.207628071308136 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9985570311546326 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03839286044239998 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1371840536594391 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.996625542640686 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09770115464925766 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 * Purity/*FRt4SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 *jet/accuracy_1_of_6‚0.9807692170143127 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16503910720348358 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35520830750465393 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 epoch‚4 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5692957043647766 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5786595940589905 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6590000987052917 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5083380341529846 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14141415059566498 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02051088772714138 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.47054973244667053 *jet/accuracy_2_of_4‚0.3506696820259094 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5266667008399963 particle/accuracy_4_of_7‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25574010610580444 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897217750549316 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011835474520921707 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5760112404823303 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5661501288414001 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6234567165374756 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18876373767852783 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11817921698093414 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9941776990890503 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2936480939388275 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5692957043647766 +particle/specificity‚0.9409050941467285 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.7083333730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32315659523010254 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3720930218696594 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6574074625968933 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.00692239822819829 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5260360240936279 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.0625 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07136532664299011 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101654887199402 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010754982009530067 )jet/accuracy_1_of_1‚0.512367844581604 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.024256346747279167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5874642729759216 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 particle/accuracy_5_of_5‚1 (particle/accuracy‚0.8371208310127258 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5274621844291687 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1383114904165268 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9597734212875366 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11783625930547714 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02058052457869053 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2331349104642868 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16120398044586182 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12814311683177948 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14262425899505615 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5215846300125122 +jet/accuracy_2_of_2‚0.10325626283884048 ,jet/accuracy_4_of_4‚0.006026997230947018 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5729286074638367 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.09502194076776505 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5185185670852661 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9823567271232605 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.12151843309402466 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.23586654663085938 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11347106099128723 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14543646574020386 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1587904840707779 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 trainer/global_step‚6044 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802686214447 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.4375 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 _step‚494 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1587904840707779 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5842559337615967 particle/accuracy_1_of_1‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_2_of_2‚1 6validation_average_jet_accuracy‚0.4495014250278473 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4721282422542572 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3854166567325592 particle/accuracy_1_of_7‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008367364294826984 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 )loss/val_total_loss‚45.46630486502299 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2384743094444275 +jet/accuracy_2_of_6‚0.47435900568962097 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_6‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05619897320866585 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.18364156782627106 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.026900924742221832 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5789473652839661 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42977678775787354 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5871469974517822 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.47986429929733276 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5313296914100647 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4616858661174774 ,jet/accuracy_5_of_5‚0.008064515888690948 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5442843437194824 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48359137773513794 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5918498635292053 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.34166666865348816 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1825396865606308 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13463521003723145 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1944444626569748 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.45075076818466187 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6333333253860474 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24645274877548218 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09794341772794724 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 'particle/f_score‚0.3461245894432068 * Purity/1FRt3SRqqt1FBt/FRt_purity‚0.375 particle/accuracy_3_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45751872658729553 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.38424691557884216 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15028735995292664 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.3892543911933899 *jet/accuracy_1_of_2‚0.7730676531791687 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.046738673001527786 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 particle/accuracy_3_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5603658556938171 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3497682511806488 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2977462410926819 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20455457270145416 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24792563915252686 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14511340856552124 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2096046507358551 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9597222805023193 particle/accuracy_4_of_4‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3083333671092987 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 particle/accuracy_5_of_6‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3778842091560364 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_5‚0.02150537632405758 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2737075984477997 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5453597903251648 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3444959223270416 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17456413805484772 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 )jet/accuracy_1_of_5‚0.965732216835022 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2509259581565857 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5470290184020996 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9854181408882141 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.00657894741743803 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39044734835624695 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5425363779067993 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6042498350143433 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.7083333730697632 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17263108491897583 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.994246780872345 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5479544401168823 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20203690230846405 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.134332537651062 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11018085479736328 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014016074128448963 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5311877727508545 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10660479217767715 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6394432187080383 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5345146656036377 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0956488624215126 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5640880465507507 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9876686930656433 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6403805613517761 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3701569736003876 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03905374929308891 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.622412383556366 +particle/sensitivity‚0.2762800455093384 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.586309552192688 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5721470713615417 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9824585318565369 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.14222222566604614 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08160187304019928 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0046040378510952 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.48725494742393494 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.977398157119751 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020674937404692173 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24197319149971008 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5151730179786682 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5892857313156128 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2806496322154999 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1158113181591034 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5948275923728943 *jet/accuracy_2_of_5‚0.4623016119003296 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.97284996509552 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.11083588004112244 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9828663468360901 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 jet/accuracy_4_of_7‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3649912178516388 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.45955169200897217 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35505053400993347 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5847222805023193 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03407225012779236 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9533045291900635 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1029883623123169 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5435823798179626 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19750964641571045 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9306416511535645 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4618476629257202 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3474701941013336 * Purity/*FRt4SRqqt*FBt/FRt_purity‚0.125 <&Pu°ßš rity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08024691045284271 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.194403275847435 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15202702581882477 particle/accuracy_1_of_2‚1 % _timestamp‚1.7884212868607614e+09î(Sß2À- _wandbruntime‚4793  _runtime‚4793¤jšÆÁÀ & lr-AdamW/pg1‚9.889342707817135e-05 & lr-AdamW/pg2‚9.889342707817135e-05 trainer/global_step‚6049 $ _timestamp‚1.788421293731656e+09 _runtime‚4793.469212086 _step‚495 ækbËÂÅ trainer/global_step‚6049 $ _timestamp‚1.788421293731656e+09 _runtime‚4793.469212086 _step‚495 & lr-AdamW/pg1‚9.889342707817135e-05 & lr-AdamW/pg2‚9.889342707817135e-05ï\$—•2Ã- _wandbruntime‚4793  _runtime‚4793wô sÝ'Ä×O OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202584847 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384938450292224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356521 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80544865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29073590 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352554238585344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390262552101888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343303207515136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51470860 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚ 4793.4701255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5074131439104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5055446477126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348358653992262 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114376559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348359325469378 trainer/global_step‚6049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384938474232320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚415 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5074787801600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _step‚496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202564807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5074131439104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5324101809664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5074787801600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430ˆ'Xâ'ÅÜO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352554256588288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88208286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚390262754345984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357628370024448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357629044389888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80544865 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29073590 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352554238585344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390262552101888 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352554238585344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29074003 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚343303207515136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88208286 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51470860 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114376559 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 _runtime‚ 4793.4701255 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5074131439104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5055446477126 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348358653992262 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114376559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348359325469378 trainer/global_step‚6049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚343303224554496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384938474232320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚415 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5074787801600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80544450 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5056100914882 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 _step‚496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202564807 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5074131439104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88208288 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5324101809664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357629044389888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5074787801600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚413 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51470862 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202584847 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚384938450292224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114356521ððŽ52Æ- _wandbruntime‚4794  _runtime‚4794,$oÚë&ÇåM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343358111645696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚497 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352610609627136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 trainer/global_step‚6049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88222302 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202597277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352610591624192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884212944098222e+09 _runtime‚4794.147984395 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51479030 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054Tvè)IÈêM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348414374557786 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5074950866944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚390325047411712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88222302 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202597277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29078086 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/alÍñWõ§"located_bytes.large_pool.freed‚352610591624192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884212944098222e+09 _runtime‚4794.147984395 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114374975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51479030 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5075607242752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5075607242752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385000101029888 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29078518 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056917350054 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚343358111645696 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114395039 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚357686216869888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202617343 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80557114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352610591624192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348415046035110 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚497 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352610609627136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202617343 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88222304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352610609627136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5325126769664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357686216869888 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88222302 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114395039 trainer/global_step‚6049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202597277 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324946381824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390325251739648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ñ"‘Ž2É- _wandbruntime‚4794  _runtime‚4794 …ï]Ê× _step‚498 /loss/FRt2/assignment_loss‚3.805899674242193 /loss/FRt2/detection_loss‚2.9650929642446116 1loss/SRqqt2/detection_loss‚2.3849106675812175 epoch‚5 1loss/FBt2/assignment_loss‚0.04038427397605915 /loss/FRt1/detection_loss‚2.2180803655223413 1loss/SRqqt1/detection_loss‚1.8377556699243458 /loss/FBt4/detection_loss‚0.7545515054439688 &loss/total_loss‚47.935032686113686 0loss/FBt1/assignment_loss‚0.5677307975129189 1loss/FBt4/assignment_loss‚0.20546017937901245 /loss/FBt3/detection_loss‚2.9193759113550186 1loss/SRqqt2/assignment_loss‚2.081544947411333 /loss/FRt3/detection_loss‚2.4480296393235523 1loss/SRqqt4/detection_loss‚1.0721118333232071 _runtime‚4794.148995763 /loss/FRt1/assignment_loss‚3.343207051808184 .loss/FBt2/detection_loss‚6.027934789657593 % _timestamp‚1.7884212944115021e+09 0loss/FRt3/assignment_loss‚2.9597782082027857 2loss/SRqqt1/assignment_loss‚1.4709049867974087 2loss/SRqqt4/assignment_loss‚1.0445107160550025 1loss/FBt3/assignment_loss‚0.33138406043872237 /loss/FRt4/detection_loss‚1.7347454510077283 trainer/global_step‚6049 /loss/FRt4/assignment_loss‚1.678608058779328 1loss/SRqqt3/detection_loss‚1.9288080036640167 /loss/FBt1/detection_loss‚2.7510470002889633 1loss/SRqqt3/assignment_loss‚1.363175930174173ô ,bËÜ /loss/FRt2/assignment_loss‚3.805899674242193 /loss/FRt2/detection_loss‚2.9650929642446116 1loss/SRqqt2/detection_loss‚2.3849106675812175 epoch‚5 1loss/FBt2/assignment_loss‚0.04038427397605915 /loss/FRt1/detection_loss‚2.2180803655223413 1loss/SRqqt1/detection_loss‚1.8377556699243458 /loss/FBt4/detection_loss‚0.7545515054439688 &loss/total_loss‚47.935032686113686 0loss/FBt1/assignment_loss‚0.5677307975129189 1loss/FBt4/assignment_loss‚0.20546017937901245 /loss/FBt3/detection_loss‚2.9193759113550186 1loss/SRqqt2/assignment_loss‚2.081544947411333 /loss/FRt3/detection_loss‚2.4480296393235523 1loss/SRqqt4/detection_loss‚1.0721118333232071 _runtime‚4794.148995763 /loss/FRt1/assignment_loss‚3.343207051808184 .loss/FBt2/detection_loss‚6.027934789657593 % _timestamp‚1.7884212944115021e+09 0loss/FRt3/assignment_loss‚2.9597782082027857 2loss/SRqqt1/assignment_loss‚1.4709049867974087 2loss/SRqqt4/assignment_loss‚1.0445107160550025 1loss/FBt3/assignment_loss‚0.33138406043872237 /loss/FRt4/detection_loss‚1.7347454510077283 trainer/global_step‚6049 /loss/FRt4/assignment_loss‚1.678608058779328 1loss/SRqqt3/detection_loss‚1.9288080036640167 /loss/FBt1/detection_loss‚2.7510470002889633 1loss/SRqqt3/assignment_loss‚1.363175930174173 _step‚498ò³}¹éœÌ‚(:‘ ´ÉäÔÅü´~ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.04' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.608000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÿËt~Í‚(:ó ´ÉäÔ­÷߀* disk./.usagePercent‚17.346306849647473 disk.rbd1.out‚2.453125& disk.nvme0n1.out‚20998.67626953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 314.41015625 disk.sda1.in‚0) proc.memory.availableMB‚ 1424.34765625 disk.nvme0c0n1.in‚ 4.33203125 network.sent‚ 1.3856181e+07# disk.dm-0.out‚20998.67626953125 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.in‚0 disk.sda.in‚ 126.77734375 network.recv‚ 1.807507e+06 disk.dm-0.in‚ 4.33203125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2772.390625 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 314.41015625 disk.rbd0.out‚0 disk.sda3.out‚ 314.41015625" memory_percent‚91.306471824646* proc.memory.percent‚16.921329498291016 cpu‚25.361676594505642% disk./.usageGB‚258.47250747680664 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚20998.67626953125 disk.sda3.in‚ 126.77734375jÀ.e’΂(:‡ ÃÉäÔøÅ“~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚166.518' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3036 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÆ’ãyÏ‚(:î ÃÉäÔÊòõ disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 315.203125 disk.dm-0.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 network.recv‚ 1.812852e+06 disk.dm-1.out‚ 315.203125 network.sent‚ 1.4001471e+07# disk.dm-0.out‚21052.67626953125 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0 cpu‚25.360668318433483 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.drbd2080.out‚0 disk.rbd1.in‚0$ memory_percent‚91.30651950836182) proc.memory.availableMB‚ 1424.33984375! proc.memory.rssMB‚ 2772.390625 disk.nvme0c0n1.in‚ 4.33203125 disk.drbd2265.out‚0 disk.sda.out‚ 315.203125 disk.drbd1716.in‚0* disk./.usagePercent‚17.348044372782358 disk.drbd2080.in‚0 disk.rbd1.out‚2.453125 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17% disk./.usageGB‚258.49839782714844& disk.nvme0n1.out‚21052.67626953125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21052.67626953125 disk.sda2.in‚0 disk.drbd2265.in‚0V˯š2Ð- _wandbruntime‚4829  _runtime‚4829Œyc-ÆÑÀ _runtime‚4829.800495412 _step‚499 & lr-AdamW/pg1‚9.769261551067274e-05 & lr-AdamW/pg2‚9.769261551067274e-05 trainer/global_step‚6099 $ _timestamp‚1.788421330062972e+09 ž?ËÒÅ trainer/global_step‚6099 $ _timestamp‚1.788421330062972e+09 _runtime‚4829.800495412 _step‚499 & lr-AdamW/pg1‚9.769261551067274e-05 & lr-AdamW/pg2‚9.769261551067274e-05ó"¼6Ð2Ó- _wandbruntime‚4829  _runtime‚4829Wà'ÔÚO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115282573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5115773607424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388020987175424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚351145367336786 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88909086 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29299116 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5366993213952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115302611 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360487682138112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393387808376832 _runtime‚ 4829.80139009 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51879262 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360487007772672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚393388004329472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚393387808376832 _runtime‚ 4829.80139009 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51879262 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346048431082496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115302611 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5366821201408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88909088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388021011115520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81177935 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51879260 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88909086 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88909088 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚6099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346103318173696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51887430 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388082637913088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚501 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚4830.486825558 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355428243566592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872dt®ð&ÚêM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346103318173696 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5115936672768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29303611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29303160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5367667586048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5097752689462 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115321091 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51887430 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204244195 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388082637913088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204224129 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88923102 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5115936672768 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360544180239360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204244195 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81190588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚501 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81191041 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚4830.486825558 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚355428243566592 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204224129 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚355428261569536 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88923104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚393450305499136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115301027 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚351201087902518 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88923104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360544854618112 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360544854618112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51887428 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5097098251498 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚351200416425194 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚6099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚355428261569536õ³6pv2Û- _wandbruntime‚4830  _runtime‚4830:Èç¶ZÜÔ /loss/FRt2/detection_loss‚2.9752166921442207 /loss/FRt1/assignment_loss‚3.821390615434063 2loss/SRqqt4/assignment_loss‚0.8246009639784027 0loss/FBt2/assignment_loss‚0.6950443035457283 /loss/FBt2/detection_loss‚2.5255008816719053 /loss/FBt3/detection_loss‚3.6349770188331605 $ _timestamp‚1.788421330750386e+09 1loss/FBt4/assignment_loss‚0.20789788489722505 .loss/FRt1/detection_loss‚2.149664731658235 2loss/SRqqt2/assignment_loss‚2.0867578999972656 1loss/SRqqt3/assignment_loss‚1.488098256899552 /loss/FRt4/detection_loss‚1.7535515219277265 1loss/SRqqt3/detection_loss‚1.9641937017440796 0loss/FRt3/assignment_loss‚2.8920874630029383 epoch‚5 0loss/FRt2/assignment_loss‚3.7945919181361343 0loss/FBt3/assignment_loss‚0.2029329399985727 0loss/FRt4/assignment_loss‚1.8363639456885201 .loss/FRt3/detection_loss‚2.301642926553121 0loss/SRqqt2/detection_loss‚2.292055661741056 1loss/SRqqt4/detection_loss‚1.2609715064181322 /loss/FBt4/detection_loss‚0.5458707352467801 _runtime‚4830.487820726 1loss/SRqqt1/assignment_loss‚1.487377677373588 _step‚502 1loss/FBt1/assignment_loss‚0.20379308350584324 1loss/SRqqt1/detection_loss‚1.8046182239055633 .loss/FBt1/detection_loss‚2.571559265255928 &loss/total_loss‚45.320759819557736 trainer/global_step‚6099gàÑ_ÝÙ 0loss/FRt4/assignment_loss‚1.8363639456885201 .loss/FRt3/detection_loss‚2.301642926553121 0loss/SRqqt2/detection_loss‚2.292055661741056 1loss/SRqqt4/detection_loss‚1.2609715064181322 /loss/FBt4/detection_loss‚0.5458707352467801 _runtime‚4830.487820726 1loss/SRqqt1/assignment_loss‚1.487377677373588 _step‚502 1loss/FBt1/assignment_loss‚0.20379308350584324 1loss/SRqqt1/detection_loss‚1.8046182239055633 .loss/FBt1/detection_loss‚2.571559265255928 &loss/total_loss‚45.320759819557736 trainer/global_step‚6099 /loss/FRt2/detection_loss‚2.9752166921442207 /loss/FRt1/assignment_loss‚3.821390615434063 2loss/SRqqt4/assignment_loss‚0.8246009639784027 0loss/FBt2/assignment_loss‚0.6950443035457283 /loss/FBt2/detection_loss‚2.5255008816719053 /loss/FBt3/detection_loss‚3.6349770188331605 $ _timestamp‚1.788421330750386e+09 1loss/FBt4/assignment_loss‚0.20789788489722505 .loss/FRt1/detection_loss‚2.149664731658235 2loss/SRqqt2/assignment_loss‚2.0867578999972656 1loss/SRqqt3/assignment_loss‚1.488098256899552 /loss/FRt4/detection_loss‚1.7535515219277265 1loss/SRqqt3/detection_loss‚1.9641937017440796 0loss/FRt3/assignment_loss‚2.8920874630029383 epoch‚5 0loss/FRt2/assignment_loss‚3.7945919181361343 0loss/FBt3/assignment_loss‚0.2029329399985727öÈ“ì€Þ‚(:’ áÉäÔÝŠ¥~ gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.631' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.126200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÇ,mß‚(:â áÉäÔßõÈ€ disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.33203125$ memory_percent‚91.30675792694092 disk.sda1.out‚0 network.sent‚ 1.4104809e+07% disk./.usageGB‚258.51036071777344 disk.drbd1716.in‚0! disk.nvme0n1.out‚ 21146.859375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2772.390625 disk.dm-1.out‚ 316.0859375 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0) proc.memory.availableMB‚ 1424.30078125 disk.rbd1.out‚2.453125 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 4.33203125# disk.nvme0c0n1.out‚ 21146.859375 disk.drbd2266.in‚0 disk.sda3.out‚ 316.0859375* proc.memory.percent‚16.921329498291016 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 21146.859375 cpu‚25.358696568719253 network.recv‚ 1.82295e+06* disk./.usagePercent‚17.348847212409737 disk.sda.out‚ 316.0859375 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375ªSˆ’à‚(:‡ ðÉäÔ‹• ~ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó')Ìuá‚(:ê ðÉäÔÖ‹­ disk.sda1.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 4.33203125 disk.drbd2266.out‚0' proc.memory.availableMB‚ 1424.296875 cpu‚25.35774055155455 network.recv‚ 1.829292e+06 disk.drbd1716.in‚0! disk.dm-0.out‚21199.052734375 disk.rbd1.out‚2.453125$ memory_percent‚91.30678176879883 proc.cpu.threads‚17$ disk.nvme0n1.out‚21199.009765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.out‚ 316.52734375 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.out‚ 316.52734375! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.4203207e+07$ disk./.usageGB‚258.5160026550293 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 4.33203125& disk.nvme0c0n1.out‚21199.009765625 disk.dm-1.in‚ 126.77734375) disk./.usagePercent‚17.34922584755288 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0* proc.memory.percent‚16.921329498291016 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 316.52734375 disk.drbd2266.in‚0ø]j·2â- _wandbruntime‚4866  _runtime‚4866átxTÇãÁ & lr-AdamW/pg1‚9.648902196653672e-05 & lr-AdamW/pg2‚9.648902196653672e-05 trainer/global_step‚6149 % _timestamp‚1.7884213663537915e+09 _runtime‚4866.091338526 _step‚503£ŒóÌäÆ & lr-AdamW/pg1‚9.648902196653672e-05 & lr-AdamW/pg2‚9.648902196653672e-05 trainer/global_step‚6149 % _timestamp‚1.7884213663537915e+09 _runtime‚4866.091338526 _step‚503÷œ­2å- _wandbruntime‚4866  _runtime‚4866'ÂæÛO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚6149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358189542470144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396513179741184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205818511 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3581Âì ¸D!89560473088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353931409204194 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348793620571136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81812081 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52287662 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5137117155942 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363346319886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844¬gìæ'çàO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚391103547998720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5409655682560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348793637610496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205838551 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52287660 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5409823500800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5156759413248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚6149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358189542470144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚358189542470144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396513179741184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚363345645520896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205818511 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358189560473088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116208625 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚353931409204194 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348793620571136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81812081 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89609888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89609886 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52287662 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5137117155942 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚363346319886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353987129769926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348848541741056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788421367085349e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚505 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89623904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5157578854400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358245895508992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116227079 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5157578854400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905ía—êéM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52295830 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137933590906 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚391165174796288 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚363402817987584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5156922478592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29528905 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52295828 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358245913511936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚6149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205871047 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29528468 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205871047 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89623904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocat?o­¤Xed‚353987129769926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89623902 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348848541741056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788421367085349e+09 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚505 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚363403492366336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358245913511936 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89623902 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81824296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89623904 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5157578854400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81824735 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205850981 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚391165198736384 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116227079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358245895508992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116227079 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5410517284864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116247143 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5157578854400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116247143 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ùõÕ™2ë- _wandbruntime‚4866  _runtime‚4866ë¹ ª[ìÕ 0loss/FRt1/assignment_loss‚2.7292960253026752 0loss/FRt4/assignment_loss‚1.6157046036842542 .loss/FRt4/detection_loss‚2.077384338547022 0loss/FBt4/assignment_loss‚0.1539931175307994 1loss/SRqqt3/detection_loss‚1.7318747983529017 /loss/FRt2/assignment_loss‚4.506986249238253 0loss/FRt3/assignment_loss‚3.2827282852056077 2loss/SRqqt2/assignment_loss‚2.1464577447623014 1loss/FBt2/assignment_loss‚0.26731867806908366 1loss/SRqqt2/detection_loss‚2.8145079811414084 /loss/FBt4/detection_loss‚0.4992380231878785 1loss/SRqqt4/assignment_loss‚1.048426815371182 /loss/FRt1/detection_loss‚2.3177070054743023 /loss/FRt2/detection_loss‚2.8887196462601423 1loss/SRqqt4/detection_loss‚0.9890616454962474 .loss/FBt2/detection_loss‚2.584712694088618 2loss/SRqqt1/assignment_loss‚1.3447889408894949 0loss/FBt1/assignment_loss‚0.5029796436429024 .loss/FBt3/detection_loss‚4.035536527633667 trainer/global_step‚6149 2loss/SRqqt3/assignment_loss‚1.2282897761593072 1loss/FBt3/assignment_loss‚0.03735603205859661 /loss/FRt3/detection_loss‚2.2024289774043218 %loss/total_loss‚45.09105399046865 % _timestamp‚1.7884213670876198e+09 _runtime‚4866.825132832 _step‚506 0loss/SRqqt1/detection_loss‚1.651654676515229 /loss/FBt1/detection_loss‚2.4339017644524574 epoch‚5Óe `íÚ 0loss/FBt1/assignment_loss‚0.5029796436429024 .loss/FBt3/detection_loss‚4.035536527633667 trainer/global_step‚6149 2loss/SRqqt3/assignment_loss‚1.2282897761593072 1loss/FBt3/assignment_loss‚0.03735603205859661 /loss/FRt3/detection_loss‚2.2024289774043218 %loss/total_loss‚45.09105399046865 % _timestamp‚1.7884213670876198e+09 _runtime‚4866.825132832 _step‚506 0loss/SRqqt1/detection_loss‚1.651654676515229 /loss/FBt1/detection_loss‚2.4339017644524574 epoch‚5 0loss/FRt1/assignment_loss‚2.7292960253026752 0loss/FRt4/assignment_loss‚1.6157046036842542 .loss/FRt4/detection_loss‚2.077384338547022 0loss/FBt4/assignment_loss‚0.1539931175307994 1loss/SRqqt3/detection_loss‚1.7318747983529017 /loss/FRt2/assignment_loss‚4.506986249238253 0loss/FRt3/assignment_loss‚3.2827282852056077 2loss/SRqqt2/assignment_loss‚2.1464577447623014 1loss/FBt2/assignment_loss‚0.26731867806908366 1loss/SRqqt2/detection_loss‚2.8145079811414084 /loss/FBt4/detection_loss‚0.4992380231878785 1loss/SRqqt4/assignment_loss‚1.048426815371182 /loss/FRt1/detection_loss‚2.3177070054743023 /loss/FRt2/detection_loss‚2.8887196462601423 1loss/SRqqt4/detection_loss‚0.9890616454962474 .loss/FBt2/detection_loss‚2.584712694088618 2loss/SRqqt1/assignment_loss‚1.3447889408894949ú½8}Iî‚(:’ ÿÉäÔÊÙš~ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÝ7vï‚(:ë ÿÉäÔÓ¬»€ disk.drbd2080.out‚0 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 317.1640625 disk.sda1.in‚0( proc.memory.availableMB‚ 1424.7265625 network.recv‚ 1.832794e+06 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚21239.361328125 network.sent‚ 1.4208633e+07 disk.dm-0.in‚ 4.33203125! disk.dm-0.out‚21239.361328125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 4.33203125* proc.memory.percent‚16.921329498291016* disk./.usagePercent‚17.352013001578154 disk.sda.out‚ 317.1640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 cpu‚25.356808164745072 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-1.out‚ 317.1640625 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2772.390625% disk./.usageGB‚258.55753326416016 disk.sda1.out‚0$ disk.nvme0n1.out‚21239.361328125$ memory_percent‚91.30415916442871 proc.cpu.threads‚17®\ï®’ð‚(:‡ ŽÊäÔž€~ gpu.0.gpu‚46.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×/Ü}ñ‚(:ò ŽÊäÔò»“€ disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.nvme0n1.in‚ 4.33203125) disk./.usagePercent‚17.35225057656993% disk./.usageGB‚258.56107330322266 disk.sda3.out‚ 317.64453125 disk.drbd2080.in‚0 disk.dm-0.in‚ 4.33203125 disk.rbd1.in‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚21304.27001953125 network.sent‚ 1.4317341e+07 disk.sda3.in‚ 126.77734375( disk.nvme0c0n1.out‚21304.27001953125 disk.sda2.out‚0* proc.memory.percent‚16.921329498291016 cpu‚25.355824656730892# disk.dm-0.out‚21304.27001953125 disk.sda1.in‚0 disk.rbd0.in‚0$ memory_percent‚94.20177936553955 network.recv‚ 1.83918e+06 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 317.64453125 disk.sda.out‚ 317.64453125 disk.sda1.out‚0! proc.memory.rssMB‚ 2772.390625 disk.drbd2266.in‚0 disk.rbd1.out‚2.453125 disk.drbd1716.out‚0( proc.memory.availableMB‚ 949.98046875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2080.out‚0¢ ¸Â2ò- _wandbruntime‚4903  _runtime‚4903R/1åÆóÀ _step‚507 & lr-AdamW/pg1‚9.528289727285385e-05 & lr-AdamW/pg2‚9.528289727285385e-05 trainer/global_step‚6199 $ _timestamp‚1.788421403369085e+09 _runtime‚4903.106619463v;½îËôÅ & lr-AdamW/pg1‚9.528289727285385e-05 & lr-AdamW/pg2‚9.528289727285385e-05 trainer/global_step‚6199 $ _timestamp‚1.788421403369085e+09 _runtime‚4903.106619463 _step‚507ûYÈÒØ2õ- _wandbruntime‚4903  _runtime‚4903hÝxà'öÚO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚6199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5197088856576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52696062 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394186060941824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 _step‚508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 4903.10756153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 Jb'÷ßO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366204957634560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5178606933106 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117134677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29749221 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207445363 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399638516548608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351538827099136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82445715 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394186084881920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117154715 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5197745219072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚434 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366204283269120 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚432 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚6199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5197088856576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚361007212415488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366204283269120 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5177952495350 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52696062 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399638702015488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394186060941824 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452617133568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚351538844138496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356717451071602 _step‚508 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52696060 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚ 4903.10756153 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361063547451392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90324704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5178768930314 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394247735619584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 trainer/global_step‚6199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361063565454336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚4903.825357601 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944d.ð&úêM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207497899 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52704230 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚356773171637334 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5179423368278 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 trainer/global_step‚6199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366261455735808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361063565454336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 _runtime‚4903.825357601 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52704228 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82457930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5453319401472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356772500160010 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚366262130114560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5198564660224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399701216534528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453480914944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207477833 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29754128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207497899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399701031080960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361063565454336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207477833 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90324704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361063547451392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚426 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _step‚509 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90324704 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5198564660224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5178768930314 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117173195 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394247735619584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90324702ý¯Kì2û- _wandbruntime‚4903  _runtime‚4903 ø6YüÓ 0loss/FBt2/assignment_loss‚0.6231351019814421 .loss/FRt2/detection_loss‚2.708314783871174 /loss/FBt4/detection_loss‚0.6420778571314105 2loss/SRqqt1/assignment_loss‚1.7227759843071302 .loss/FBt1/detection_loss‚2.117084365624648 /loss/FBt3/detection_loss‚3.2211067335946217 0loss/FRt2/assignment_loss‚3.5822482369840145 0loss/FRt4/assignment_loss‚1.4033836733017648 2loss/SRqqt2/assignment_loss‚1.8968131364353242 2loss/SRqqt3/assignment_loss‚1.4645636308877195 1loss/SRqqt2/detection_loss‚2.4399287450698113 trainer/global_step‚6199 /loss/FRt1/assignment_loss‚3.049738049507141 2loss/SRqqt4/assignment_loss‚1.0225193741019158 /loss/FBt3/assignment_loss‚0.355137494473768 /loss/FRt1/detection_loss‚2.2710604828137617 /loss/FRt3/detection_loss‚2.2575389536527486 0loss/SRqqt4/detection_loss‚1.025518303298775 epoch‚5 1loss/FBt4/assignment_loss‚0.20879056031279514 .loss/FRt4/detection_loss‚1.578812857025436 1loss/SRqqt1/detection_loss‚1.9827980094485813 _runtime‚4903.826336738 0loss/FRt3/assignment_loss‚2.8493444438164053 %loss/total_loss‚43.38028153330693 % _timestamp‚1.7884214040888908e+09 _step‚510 0loss/FBt1/assignment_loss‚0.3211609601902847 1loss/SRqqt3/detection_loss‚1.6784888593876948 .loss/FBt2/detection_loss‚2.957940936088562¡™ã–^ýØ 0loss/FBt2/assignment_loss‚0.6231351019814421 .loss/FRt2/detection_loss‚2.708314783871174 /loss/FBt4/detection_loss‚0.6420778571314105 2loss/SRqqt1/assignment_loss‚1.7227759843071302 .loss/FBt1/detection_loss‚2.117084365624648 /loss/FBt3/detection_loss‚3.2211067335946217 0loss/FRt2/assignment_loss‚3.5822482369840145 0loss/FRt4/assignment_loss‚1.4033836733017648 2loss/SRqqt2/assignment_loss‚1.8968131364353242 2loss/SRqqt3/assignment_loss‚1.4645636308877195 1loss/SRqqt2/detection_loss‚2.4399287450698113 trainer/global_step‚6199 /loss/FRt1/assignment_loss‚3.049738049507141 2loss/SRqqt4/assignment_loss‚1.0225193741019158 /loss/FBt3/assignment_loss‚0.355137494473768 /loss/FRt1/detection_loss‚2.2710604828137617 /loss/FRt3/detection_loss‚2.2575389536527486 0loss/SRqqt4/detection_loss‚1.025518303298775 epoch‚5 1loss/FBt4/assignment_loss‚0.20879056031279514 .loss/FRt4/detection_loss‚1.578812857025436 1loss/SRqqt1/detection_loss‚1.9827980094485813 _runtime‚4903.826336738 0loss/FRt3/assignment_loss‚2.8493444438164053 %loss/total_loss‚43.38028153330693 % _timestamp‚1.7884214040888908e+09 _step‚510 0loss/FBt1/assignment_loss‚0.3211609601902847 1loss/SRqqt3/detection_loss‚1.6784888593876948 .loss/FBt2/detection_loss‚2.957940936088562þ\ºã-’þ‚(:‡ ÊäÔ«è”~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6894 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö}¤}ÿ‚(:ò ÊäÔ”Þƒ disk.nvme0c0n1.in‚ 4.33203125! proc.memory.rssMB‚ 2772.390625 cpu‚25.354935689033788 network.recv‚ 1.843166e+06 disk.sda.in‚ 126.77734375 disk.sda.out‚ 318.10546875' disk.nvme0c0n1.out‚21403.1826171875 disk.drbd2265.in‚0$ memory_percent‚92.16177463531494% disk./.usageGB‚258.56514739990234 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.sda2.out‚0" disk.dm-0.out‚21403.1826171875 disk.rbd1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1284.21484375* proc.memory.percent‚16.921329498291016 network.sent‚ 1.4323242e+07 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚21403.1826171875 disk.sda3.out‚ 318.10546875 disk.dm-0.in‚ 4.33203125 disk.nvme0n1.in‚ 4.33203125 disk.sda2.in‚0 disk.rbd1.out‚2.453125* disk./.usagePercent‚17.352523992616497 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 disk.dm-1.out‚ 318.10546875 disk.drbd1716.out‚00?’€‚(:‡ ¬ÊäÔˆäú} gpu.0.gpu‚38.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¯1Üm‚(:â ¬ÊäÔ‰€ñ€ proc.cpu.threads‚17 network.sent‚ 1.4432336e+07 disk.drbd2265.in‚0 disk.dm-0.out‚21450.23046875 disk.rbd1.in‚0 disk.rbd1.out‚2.453125 network.recv‚ 1.850159e+06 disk.sda2.out‚0# disk.nvme0n1.out‚21450.23046875 disk.rbd0.in‚0* proc.memory.percent‚16.921329498291016 disk.dm-0.in‚ 4.33203125! proc.memory.rssMB‚ 2772.390625) disk./.usagePercent‚17.35285014621512% disk./.usageGB‚258.57000732421875$ memory_percent‚92.16175079345703& proc.memory.availableMB‚ 1284.21875 cpu‚25.353961916581508 disk.dm-1.out‚ 318.578125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚ 4.33203125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 318.578125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚21450.23046875 disk.sda3.out‚ 318.578125 disk.drbd2266.in‚0 disk.drbd1716.out‚0Íð’‚‚(:‡ »ÊäÔþ˜‹~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.4884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÓÒÒmƒ‚(:â »ÊäÔ€Þÿ disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚2.453125 disk.nvme0n1.in‚ 4.33203125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 319.08203125* proc.memory.percent‚16.921329498291016 disk.sda.in‚ 126.77734375 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 21502.09375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 4.33203125 cpu‚25.353001821863838 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375* disk./.usagePercent‚17.353617912810307 disk.sda.out‚ 319.08203125 disk.sda3.out‚ 319.08203125 disk.drbd2265.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.4437619e+07! proc.memory.rssMB‚ 2772.390625% disk./.usageGB‚258.58144760131836 disk.nvme0n1.out‚ 21502.09375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 4.33203125 disk.dm-0.out‚ 21502.09375$ memory_percent‚92.16177463531494) proc.memory.availableMB‚ 1284.21484375 network.recv‚ 1.853637e+06gŠð2„- _wandbruntime‚4939  _runtime‚4939MCžÇ…Á trainer/global_step‚6249 % _timestamp‚1.7884214399181795e+09 _runtime‚4939.655770489 _step‚511 & lr-AdamW/pg1‚9.407449278420254e-05 & lr-AdamW/pg2‚9.407449278420254e-05ù'ÖâÌ†Æ & lr-AdamW/pg2‚9.407449278420254e-05 trainer/global_step‚6249 % _timestamp‚1.7884214399181795e+09 _runtime‚4939.655770489 _step‚511 & lr-AdamW/pg1‚9.407449278420254e-05ÿýŒº2‡- _wandbruntime‚4939  _runtime‚4939j „ˆÛO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29974862 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5218787834758 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118060729 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397268597825024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5495498032640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402764281324544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402764095857664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397268621765120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118080767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118060729 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53104460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚354284050666496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369063595382784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5238074662400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397268597825024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5495498032640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5495659559424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚402764281324544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53104462 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚359503492939010 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚359502821461894 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118080767 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91011486 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402764095857664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397268621765120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118080767 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29974422 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369062921017344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363824846354944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5238074662400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83079324 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363824846354944 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 _step‚513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118099247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91025504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29979341 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5220258707686 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209124751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844±¯shŒèM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83091529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5496350288896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚359558542027418 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29978901 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5219604269722 _step‚513 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5239550466048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5239550466048 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118099247 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112630 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209104685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83091971 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5496520190976 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91025502 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118079183 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91025504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369120767862784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363881217396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5238894090240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29979341 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚359559213504742 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118099247 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91025504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚402826792693760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚402826598851584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363881199393792 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209104685 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5220258707686 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91025502 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H:DeviceStatsMonitor.on_train_batch_end/active.allçºÎî .allocated‚ 209124751 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363881199393792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397330248562688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369120767862784 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209124751 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5279060468224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788421477052283e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118986781 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118986781 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53512862 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119006819 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5538490378240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400351158648320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371921558765568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91712288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚371922233131008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405889625086464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400351134708224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210719107 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91712286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366642516300288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405889808456192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119006819 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚516 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚371921558765568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚6299 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5279060468224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788421477052283e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118986781 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118986781 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53512862 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5259623174166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5279060468224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83712153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119006819 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5538490378240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚400351158648320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚371921558765568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91712288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30199729 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚371922233131008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405889625086464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚400351134708224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210719107 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91712286 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚366642516300288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚362288863329302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405889808456192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210699067 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119006819 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5538649807872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚516 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceyËStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5260439609130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚517 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210731537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884214777397194e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5280536271872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357084144285696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366698851336192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚4977.477900784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357084161325056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91726302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2®LŠ8ð&œêM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5279879896064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83724750 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚6299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210751603 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405952130734592 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53521030 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362344583894826 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91726302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5260439609130 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83725186 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _step‚517 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚400412809385984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚366698869339136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210731537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405952322479616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5539513093632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5539345288704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119005235 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚362345255372150 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚371978731232256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119025299 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 % _timestamp‚1.7884214777397194e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚366698869339136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5280536271872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357084144285696 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚366698851336192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚371979405611008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91726304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _runtime‚4977.477900784 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91726304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119025299 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357084161325056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5261094047094 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91726302 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5279879896064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚371979405611008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5280536271872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚400412785445888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210731537 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210751603 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30204156…ȳÀÎ2- _wandbruntime‚4977  _runtime‚4977¶ÃzYžÓ 2loss/SRqqt1/assignment_loss‚1.7317078241280146 1loss/SRqqt3/assignment_loss‚1.849807906570652 .loss/FBt4/detection_loss‚0.525219692773887 epoch‚5 1loss/FBt2/assignment_loss‚0.19890934751128006 1loss/FBt4/assignment_loss‚0.27886540676842914 1loss/SRqqt1/detection_loss‚1.9360344400819467 1loss/SRqqt4/detection_loss‚1.1013548616049946 _step‚518 0loss/FBt1/assignment_loss‚0.5343474530511432 .loss/FRt2/detection_loss‚3.037865350323339 0loss/FRt3/assignment_loss‚2.5582598946532427 2loss/SRqqt2/assignment_loss‚2.1811690191427866 0loss/FBt3/assignment_loss‚0.7115245761812535 /loss/FBt3/detection_loss‚2.3996558700289046 0loss/SRqqt2/detection_loss‚2.458231885234515 .loss/FBt2/detection_loss‚3.095759997765223 _runtime‚4977.479135144 0loss/FRt1/assignment_loss‚3.4551311183948905 /loss/FRt3/detection_loss‚1.9913536450084375 $ _timestamp‚1.788421477741382e+09 &loss/total_loss‚44.193894053823556 trainer/global_step‚6299 /loss/FRt2/assignment_loss‚4.017800796416498 0loss/FRt4/assignment_loss‚1.3174225021811092 2loss/SRqqt4/assignment_loss‚0.8008130007436958 .loss/FRt4/detection_loss‚1.494766936874857 1loss/SRqqt3/detection_loss‚1.5735448246437407 .loss/FBt1/detection_loss‚2.765438993771871 /loss/FRt1/detection_loss‚2.1789087099688396¨˜\^ŸØ 2loss/SRqqt1/assignment_loss‚1.7317078241280146 1loss/SRqqt3/assignment_loss‚1.849807906570652 .loss/FBt4/detection_loss‚0.525219692773887 epoch‚5 1loss/FBt2/assignment_loss‚0.19890934751128006 1loss/FBt4/assignment_loss‚0.27886540676842914 1loss/SRqqt1/detection_loss‚1.9360344400819467 1loss/SRqqt4/detection_loss‚1.1013548616049946 _step‚518 0loss/FBt1/assignment_loss‚0.5343474530511432 .loss/FRt2/detection_loss‚3.037865350323339 0loss/FRt3/assignment_loss‚2.5582598946532427 2loss/SRqqt2/assignment_loss‚2.1811690191427866 0loss/FBt3/assignment_loss‚0.7115245761812535 /loss/FBt3/detection_loss‚2.3996558700289046 0loss/SRqqt2/detection_loss‚2.458231885234515 .loss/FBt2/detection_loss‚3.095759997765223 _runtime‚4977.479135144 0loss/FRt1/assignment_loss‚3.4551311183948905 /loss/FRt3/detection_loss‚1.9913536450084375 $ _timestamp‚1.788421477741382e+09 &loss/total_loss‚44.193894053823556 trainer/global_step‚6299 /loss/FRt2/assignment_loss‚4.017800796416498 0loss/FRt4/assignment_loss‚1.3174225021811092 2loss/SRqqt4/assignment_loss‚0.8008130007436958 .loss/FRt4/detection_loss‚1.494766936874857 1loss/SRqqt3/detection_loss‚1.5735448246437407 .loss/FBt1/detection_loss‚2.765438993771871 /loss/FRt1/detection_loss‚2.1789087099688396†Mzb’ ‚(:‡ èÊäÔ¢ú†~ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ÆOþx¡‚(:í èÊäÔâÉœ€ disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0' disk./.usagePercent‚17.356673618739# disk.dm-0.out‚21702.26806640625 disk.dm-1.out‚ 321.12890625 disk.sda1.out‚0& disk.nvme0n1.out‚21702.26806640625$ memory_percent‚90.65942764282227 cpu‚25.350325705594702 disk.rbd1.out‚2.453125 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.out‚ 321.12890625 disk.rbd0.in‚0! proc.memory.rssMB‚ 2772.390625* proc.memory.percent‚16.921329498291016 network.sent‚ 1.4558638e+07 disk.nvme0c0n1.in‚ 4.6015625' proc.memory.availableMB‚ 1530.359375 proc.cpu.threads‚17 disk.drbd1716.out‚0 network.recv‚ 1.869307e+06% disk./.usageGB‚258.62697982788086 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 321.12890625 disk.drbd2265.in‚0 disk.dm-0.in‚ 4.6015625 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 4.6015625 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21702.26806640625rãðÈ’¢‚(:‡ ÷ÊäÔ웹~ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.939' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.1878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¡qÇu£‚(:ê ÷ÊäÔô˜Ò€# disk.dm-0.out‚21750.08056640625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚21750.08056640625 disk.sda2.in‚0 disk.sda3.out‚ 322.03515625 network.recv‚ 1.87464e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.65625 disk.sda2.out‚0% disk./.usageGB‚258.61672592163086 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.349424123077387) disk./.usagePercent‚17.35598547048696 disk.dm-0.in‚4.65625 disk.drbd2266.in‚0 disk.sda.out‚ 322.03515625 disk.sda3.in‚ 126.77734375$ memory_percent‚90.65639972686768* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.65625 disk.dm-1.out‚ 322.03515625 disk.rbd1.out‚2.453125 disk.rbd0.in‚0 network.sent‚ 1.4666581e+07& disk.nvme0n1.out‚21750.08056640625! proc.memory.rssMB‚ 2772.390625 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375) proc.memory.availableMB‚ 1530.85546875 âÃ’¤‚(:‡ †ËäÔ¼ë¥~ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ó r¥‚(:ç †ËäÔñÅÀ€ network.sent‚ 1.4672682e+07 network.recv‚ 1.878893e+06* disk./.usagePercent‚17.358071675883487 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0# disk.dm-0.out‚21811.37841796875 disk.sda1.out‚0$ memory_percent‚90.65802097320557( disk.nvme0c0n1.out‚21811.37841796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17 disk.sda1.in‚0% disk./.usageGB‚258.64781188964844 disk.sda3.out‚ 322.859375 disk.nvme0n1.in‚4.65625& disk.nvme0n1.out‚21811.37841796875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.65625 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.6015625 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.65625 disk.dm-1.out‚ 322.859375) proc.memory.availableMB‚ 1530.58984375 disk.rbd1.in‚0* proc.memory.percent‚16.921329498291016 cpu‚25.348533759551486 disk.sda.out‚ 322.859375zŠÝQ2¦- _wandbruntime‚5015  _runtime‚5015^rǧÁ & lr-AdamW/pg1‚9.165185216335641e-05 & lr-AdamW/pg2‚9.165185216335641e-05 trainer/global_step‚6349 % _timestamp‚1.7884215156878011e+09 _runtime‚5015.425353097 _step‚519¶;ØÌ¨Æ _step‚519 & lr-AdamW/pg1‚9.165185216335641e-05 & lr-AdamW/pg2‚9.165185216335641e-05 trainer/global_step‚6349 % _timestamp‚1.7884215156878011e+09 _runtime‚5015.425353097‡Ÿ&Pê2©- _wandbruntime‚5015  _runtime‚50154ìNªÛO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374780870879232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5320046274048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30425142 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040Zvœæ'«àO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92413088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92413086 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84345956 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212325919 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409015103118336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365074905196710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365075576673826 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53921260 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84346404 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119932871 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92413086 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359774446683136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚520 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374780870879232 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119932871 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5320046274048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359774463722496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119912833 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5300458513574 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409014911360000 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403433695531520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6349 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374780196513792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5320702636544 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92413088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30424696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚403433671591424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374780196513792 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369460168242688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374780870879232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5320046274048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53921262 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5301112951330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30425142 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365131297239558 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119931287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚6349 _step‚521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369516503278592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403495322329088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5016.152287971 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5301274948538 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374837368980480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403495322329088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5016.152287971 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212358389 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409077429504000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84358607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5321522077696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5321522077696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212358389 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409077612860416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365131297239558 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92427104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92427102 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212378455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53929430 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365130625762234 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119951351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403495346269184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369516521281536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5320865701888 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5582266591232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119931287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53929428 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374838043359232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374838043359232 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚6349 _step‚521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5320865701888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92427102 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5582107174912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359829350813696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369516503278592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5301929386502 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119931287 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.larg6ù÷@je_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212378455‰ÔÂI2¯- _wandbruntime‚5016  _runtime‚5016ö0Â×U°Ï /loss/FRt1/assignment_loss‚2.932014475850498 0loss/FRt3/assignment_loss‚2.7568330940078285 0loss/FRt4/assignment_loss‚1.6605774004284928 epoch‚5 2loss/SRqqt2/assignment_loss‚2.0917678233236074 1loss/SRqqt4/assignment_loss‚1.172975559260637 1loss/FBt4/assignment_loss‚0.13846555273933622 % _timestamp‚1.7884215164163408e+09 1loss/SRqqt3/detection_loss‚1.4450188717421364 _runtime‚5016.153854187 1loss/SRqqt1/assignment_loss‚1.464766340330243 2loss/SRqqt3/assignment_loss‚1.4211664118201417 /loss/FRt3/detection_loss‚2.1846930393401314 0loss/FRt2/assignment_loss‚3.4055102325621105 0loss/FBt1/assignment_loss‚0.4934900007210672 0loss/FBt3/assignment_loss‚0.7499728503637015 0loss/SRqqt1/detection_loss‚2.038386143743992 .loss/FBt1/detection_loss‚2.472192794084549 .loss/FBt2/detection_loss‚2.970675517212261 /loss/FBt4/detection_loss‚0.7200694700130766 %loss/total_loss‚45.08841039469013 /loss/FRt1/detection_loss‚2.1970207270453956 .loss/FRt2/detection_loss‚3.308227601505461 .loss/FRt4/detection_loss‚2.116698957343654 1loss/SRqqt4/detection_loss‚1.0600026123343953 trainer/global_step‚6349 .loss/FBt2/assignment_loss‚0.80247209625363 0loss/SRqqt2/detection_loss‚2.525936149060726 /loss/FBt3/detection_loss‚2.9594766736030578 _step‚5224Ì&Z±Ô 1loss/SRqqt3/detection_loss‚1.4450188717421364 _runtime‚5016.153854187 1loss/SRqqt1/assignment_loss‚1.464766340330243 2loss/SRqqt3/assignment_loss‚1.4211664118201417 /loss/FRt3/detection_loss‚2.1846930393401314 0loss/FRt2/assignment_loss‚3.4055102325621105 0loss/FBt1/assignment_loss‚0.4934900007210672 0loss/FBt3/assignment_loss‚0.7499728503637015 0loss/SRqqt1/detection_loss‚2.038386143743992 .loss/FBt1/detection_loss‚2.472192794084549 .loss/FBt2/detection_loss‚2.970675517212261 /loss/FBt4/detection_loss‚0.7200694700130766 %loss/total_loss‚45.08841039469013 /loss/FRt1/detection_loss‚2.1970207270453956 .loss/FRt2/detection_loss‚3.308227601505461 .loss/FRt4/detection_loss‚2.116698957343654 1loss/SRqqt4/detection_loss‚1.0600026123343953 trainer/global_step‚6349 .loss/FBt2/assignment_loss‚0.80247209625363 0loss/SRqqt2/detection_loss‚2.525936149060726 /loss/FBt3/detection_loss‚2.9594766736030578 _step‚522 /loss/FRt1/assignment_loss‚2.932014475850498 0loss/FRt3/assignment_loss‚2.7568330940078285 0loss/FRt4/assignment_loss‚1.6605774004284928 epoch‚5 2loss/SRqqt2/assignment_loss‚2.0917678233236074 1loss/SRqqt4/assignment_loss‚1.172975559260637 1loss/FBt4/assignment_loss‚0.13846555273933622 % _timestamp‚1.7884215164163408e+09Šv^.š’²‚(:‡ •ËäÔ¸‹¹~ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?eƒIt³‚(:é •ËäÔë…Ü€ proc.cpu.threads‚17 disk.rbd0.out‚0$ disk./.usageGB‚258.6137161254883 disk.rbd1.out‚ 2.6015625 disk.drbd2265.out‚0 disk.sda.out‚ 323.31640625 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.921329498291016 disk.dm-1.out‚ 323.31640625 cpu‚25.3476449378693 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1530.77734375 disk.nvme0c0n1.in‚4.65625( disk.nvme0c0n1.out‚21860.51123046875* disk./.usagePercent‚17.355783480542446 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.65625 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.65625# disk.dm-0.out‚21860.51123046875 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0& disk.nvme0n1.out‚21860.51123046875 disk.sda2.in‚0! proc.memory.rssMB‚ 2772.390625 disk.sda3.out‚ 323.31640625 network.sent‚ 1.467857e+07 network.recv‚ 1.882726e+06 disk.drbd1716.out‚0$ memory_percent‚90.65687656402588úŠ'4’´‚(:‡ ¤ËäÔßÔ˜~ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜K‘¾xµ‚(:í ¤ËäÔó˜Ë€ disk.sda1.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 1530.0390625 disk.dm-1.in‚ 126.77734375 disk.nvme0n1.in‚4.65625 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17 network.sent‚ 1.4787604e+07 disk.dm-1.out‚ 323.94140625 disk.rbd1.out‚ 2.6337890625 disk.drbd2080.in‚0 network.recv‚ 1.88948e+06 disk.dm-0.in‚4.65625# disk.dm-0.out‚21912.80029296875 disk.rbd1.in‚0 disk.sda.out‚ 323.94140625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚4.65625( disk.nvme0c0n1.out‚21912.80029296875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0# memory_percent‚90.6613826751709 cpu‚25.346765029301764 disk.drbd1716.in‚0 disk.sda3.out‚ 323.94140625* disk./.usagePercent‚17.357542764317742% disk./.usageGB‚258.63993072509766 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.drbd2266.in‚0& disk.nvme0n1.out‚21912.80029296875 disk.rbd0.out‚0* proc.memory.percent‚16.921329498291016hœ—­2¶- _wandbruntime‚5054  _runtime‚5054®+¡°Ç·Á _step‚523 & lr-AdamW/pg1‚9.043812090583584e-05 & lr-AdamW/pg2‚9.043812090583584e-05 trainer/global_step‚6399 % _timestamp‚1.7884215543870583e+09 _runtime‚5054.124585718’Ÿ`Ì¸Æ trainer/global_step‚6399 % _timestamp‚1.7884215543870583e+09 _runtime‚5054.124585718 _step‚523 & lr-AdamW/pg1‚9.043812090583584e-05 & lr-AdamW/pg2‚9.043812090583584e-05‹02¹- _wandbruntime‚5054  _runtime‚5054ö‰Évá'ºÛO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406516232414720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120858923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84980059 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84979616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93113888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54329662 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372277802182144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362519670250496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5341948290738 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚524 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93113888 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚377639508627456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884215543878615e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372277820185088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5341293852982 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120838885 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5361688442368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30650397 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5624164002304 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120858923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372277820185088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93113886 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚377638834262016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚367861618541234 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚6399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84980059 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5624319237632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213972811 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84979616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367860947064118 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5361032079872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93113888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377638834262016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54329660 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54329662 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5361688442368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412140551652352 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372277802182144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362519670250496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213952771 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372277802182144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361032079872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5341948290738 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚524Œ©±É2¼- _wandbruntime‚5054  _runtime‚5054Â+E½äM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚6399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362574557341696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5342110287946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362574574381056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5054.840772021 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batchb‹¥_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚525 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367917339106966 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377696006728704 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872ú°÷ï&¾éM PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84992245 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93127904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213985241 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120857339 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚525 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367917339106966 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214005307 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377696006728704 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214005307 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120857339 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30654417 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412203067332608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93127904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372334173223936 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372334155220992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 213985241 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚6399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚372334173223936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5362507883520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30654860 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367916667629642 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362574557341696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5342110287946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120877403 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚362574574381056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54337828 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5054.840772021eÞ(ƒ2¿- _wandbruntime‚5054  _runtime‚5054µ‹WYÀÓ /loss/FBt3/detection_loss‚2.3899777105876376 trainer/global_step‚6399 1loss/FBt4/assignment_loss‚0.15700825828542558 1loss/SRqqt2/detection_loss‚2.3262888896159635 0loss/SRqqt3/detection_loss‚1.671578271711459 0loss/SRqqt1/detection_loss‚2.022221323974589 /loss/FBt2/detection_loss‚2.1745994637409845 0loss/FRt1/assignment_loss‚3.4808964084934546 /loss/FRt3/detection_loss‚2.1319779025183783 _runtime‚ 5054.84179454 /loss/FBt4/detection_loss‚0.5939926794584985 /loss/FRt3/assignment_loss‚2.417254043949975 1loss/SRqqt2/assignment_loss‚1.736040594104009 2loss/SRqqt4/assignment_loss‚1.0796627020463347 1loss/FBt2/assignment_loss‚0.22228085735696368 1loss/SRqqt4/detection_loss‚1.0281666906221825 0loss/FRt2/assignment_loss‚3.7323728674336483 0loss/FBt3/assignment_loss‚0.2501817601067679 .loss/FRt1/detection_loss‚2.358082702836475 /loss/FRt2/detection_loss‚2.7659650774378526 0loss/FRt4/assignment_loss‚0.9499241626053526 2loss/SRqqt3/assignment_loss‚1.5258084757284063 .loss/FRt4/detection_loss‚1.636294930192985 .loss/FBt1/detection_loss‚4.266628662745158 $ _timestamp‚1.788421555104348e+09 _step‚526 2loss/SRqqt1/assignment_loss‚1.5010910410671792 0loss/FBt1/assignment_loss‚0.6494781735042731 &loss/total_loss‚43.067773650123954 epoch‚52O‚%^ÁØ 0loss/FRt1/assignment_loss‚3.4808964084934546 /loss/FRt3/detection_loss‚2.1319779025183783 _runtime‚ 5054.84179454 /loss/FBt4/detection_loss‚0.5939926794584985 /loss/FRt3/assignment_loss‚2.417254043949975 1loss/SRqqt2/assignment_loss‚1.736040594104009 2loss/SRqqt4/assignment_loss‚1.0796627020463347 1loss/FBt2/assignment_loss‚0.22228085735696368 1loss/SRqqt4/detection_loss‚1.0281666906221825 0loss/FRt2/assignment_loss‚3.7323728674336483 0loss/FBt3/assignment_loss‚0.2501817601067679 .loss/FRt1/detection_loss‚2.358082702836475 /loss/FRt2/detection_loss‚2.7659650774378526 0loss/FRt4/assignment_loss‚0.9499241626053526 2loss/SRqqt3/assignment_loss‚1.5258084757284063 .loss/FRt4/detection_loss‚1.636294930192985 .loss/FBt1/detection_loss‚4.266628662745158 $ _timestamp‚1.788421555104348e+09 _step‚526 2loss/SRqqt1/assignment_loss‚1.5010910410671792 0loss/FBt1/assignment_loss‚0.6494781735042731 &loss/total_loss‚43.067773650123954 epoch‚5 /loss/FBt3/detection_loss‚2.3899777105876376 trainer/global_step‚6399 1loss/FBt4/assignment_loss‚0.15700825828542558 1loss/SRqqt2/detection_loss‚2.3262888896159635 0loss/SRqqt3/detection_loss‚1.671578271711459 0loss/SRqqt1/detection_loss‚2.022221323974589 /loss/FBt2/detection_loss‚2.1745994637409845Ž¸÷l͒‚(:‡ ³ËäÔ ˜¢~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ—q‰xÂ(:í ³ËäÔž¸Î€( disk.nvme0c0n1.out‚21953.11279296875 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.65625! proc.memory.rssMB‚ 2772.390625$ disk./.usageGB‚258.6084098815918 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 324.47265625 disk.rbd1.out‚ 2.6337890625 disk.sda.out‚ 324.47265625 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17# disk.dm-0.out‚21953.11279296875 disk.sda.in‚ 126.77734375 disk.sda2.in‚0$ memory_percent‚90.65713882446289* proc.memory.percent‚16.921329498291016 disk.drbd2080.in‚0 disk.dm-0.in‚4.65625' proc.memory.availableMB‚ 1530.734375 network.recv‚ 1.893052e+06 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚21953.11279296875 disk.sda3.out‚ 324.47265625 network.sent‚ 1.4792921e+07* disk./.usagePercent‚17.355427374062316 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.65625 cpu‚25.345888752319016 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0  íÄ‚(:… ÂËäÔÏ¥€~ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.438 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 -ÇRxÅ‚(:í ÂËäÔ³ô…€ disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.69677734375 disk.sda.out‚ 325.16015625 disk.drbd1716.in‚0% disk.nvme0n1.out‚22007.7705078125' disk.nvme0c0n1.out‚22007.7705078125 network.recv‚ 1.898743e+06 disk.drbd2080.out‚0 disk.dm-0.in‚4.65625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 325.16015625 network.sent‚ 1.4900877e+07) disk./.usagePercent‚17.35796568876431% disk./.usageGB‚258.64623260498047 disk.drbd2265.out‚0$ memory_percent‚90.65711498260498! proc.memory.rssMB‚ 2772.390625 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.65625* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 disk.drbd2266.in‚0 cpu‚25.345072730807882 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.65625 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 325.16015625 disk.rbd0.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1530.73828125" disk.dm-0.out‚22007.7705078125 disk.sda2.out‚0Âïo5œÆ‚(:‘ ÑËäÔǤ­~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.693999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{9jœvÇ‚(:ë ÑËäÔÒÂà€ network.sent‚ 1.4906703e+07 disk.rbd0.out‚0" disk.dm-0.out‚22038.3818359375 disk.dm-1.out‚ 325.76953125 disk.sda.out‚ 325.76953125 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1530.71484375 disk.drbd2266.in‚0* proc.memory.percent‚16.921329498291016 disk.sda3.out‚ 325.76953125% disk.nvme0n1.out‚22038.3818359375' disk.nvme0c0n1.out‚22038.3818359375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.nvme0n1.in‚4.65625 disk.sda2.out‚0 disk.dm-0.in‚4.65625 disk.drbd1716.in‚0) disk./.usagePercent‚17.35912258681154$ disk./.usageGB‚258.6634712219238 disk.sda2.in‚0 disk.rbd1.out‚ 2.69677734375$ memory_percent‚90.65725803375244 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.65625 disk.sda1.out‚0! proc.memory.rssMB‚ 2772.390625 network.recv‚ 1.902936e+06 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 cpu‚25.34419943063014N|a2È- _wandbruntime‚5092  _runtime‚5092ÉE°ÿÅÉ¿ & lr-AdamW/pg1‚8.922311949748021e-05 & lr-AdamW/pg2‚8.922311949748021e-05 trainer/global_step‚6449 # _timestamp‚1.78842159311589e+09 _runtime‚5092.853446129 _step‚527“St©ÊÊÄ & lr-AdamW/pg1‚8.922311949748021e-05 & lr-AdamW/pg2‚8.922311949748021e-05 trainer/global_step‚6449 # _timestamp‚1.78842159311589e+09 _runtime‚5092.853446129 _step‚527: øX2Ë- _wandbruntime‚5092  _runtime‚5092á-_Œà'ÌÚO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884215931166425e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85614579 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54738060 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5402017885696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409598745357824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121784975 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54738062 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30876965 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5382783630146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5402674248192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚6449 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667260854272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5382129192390 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 5092.85443771 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380498146375680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215599663 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5402674248192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365264876778496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928®y-¼ÍßO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5382783630146 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NÕ±±:Ð&@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚375095454124544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5402674248192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚6449 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121764937 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215579623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚409598769297920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121784975 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚415265836296704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5667260854272 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30876519 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚370647660408642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93814686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5667090938880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5382129192390 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 5092.85443771 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚365264859739136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚415266030152192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380497472010240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚380498146375680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380498146375680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚370646988931526 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5402017885696 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85615027 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215599663 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5402674248192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚375095472127488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚365264876778496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚375095472127488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884215931166425e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93814688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85614579 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54738060 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5402017885696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚409598745357824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380497472010240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚375095454124544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121784975 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93814688 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121764937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54738062 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93814686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30876965 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414v™3•2Î- _wandbruntime‚5093  _runtime‚5093¯´‰»ë&ÏåM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121803455 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚415328341186048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚529 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5383600065318 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30881446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121783391 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85627676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884215938552525e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370703380974374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93828704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚380555318855680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93828704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884215938552525e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667945090560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215632159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚370703380974374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5402837313536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93828704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85627226 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215632159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚409660420035584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚529 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚380554644476928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚375151825166336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚375151807163392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5402837313536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215612093 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚409660396095488 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5383600065318 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54746230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30881446 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215612093 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121783391 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85627676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066‘îé¶2Ñ- _wandbruntime‚5093  _runtime‚5093½HTZÒÔ 1loss/FBt1/assignment_loss‚0.20413777008652687 % _timestamp‚1.7884215938574188e+09 _runtime‚5093.594892879 /loss/FRt4/assignment_loss‚1.612547021739337 1loss/SRqqt3/detection_loss‚1.4537898128586155 0loss/FRt2/assignment_loss‚3.0464604551141914 0loss/FRt3/assignment_loss‚2.8032632280479777 /loss/FRt1/detection_loss‚2.3837658922339595 -loss/FRt3/detection_loss‚2.22051500100078 0loss/SRqqt4/detection_loss‚1.091165843271259 /loss/FBt1/detection_loss‚1.8821014493703843 .loss/FBt2/detection_loss‚3.770610673086984 trainer/global_step‚6449 1loss/SRqqt1/detection_loss‚1.7835732246083873 &loss/total_loss‚41.378681064787656 1loss/FBt4/assignment_loss‚0.12053801095922526 /loss/FRt4/detection_loss‚1.5014493653208625 epoch‚5 /loss/FRt1/assignment_loss‚3.127511907455533 2loss/SRqqt2/assignment_loss‚1.7922347411513329 2loss/SRqqt4/assignment_loss‚0.7260107439132812 1loss/FBt2/assignment_loss‚0.29869432218583825 /loss/SRqqt2/detection_loss‚2.41551753713025 _step‚530 2loss/SRqqt3/assignment_loss‚1.4985260561640774 1loss/FBt3/assignment_loss‚0.32030188851058483 .loss/FRt2/detection_loss‚2.819441228201895 /loss/FBt3/detection_loss‚2.4050681442022324 /loss/FBt4/detection_loss‚0.7616167866280708 1loss/SRqqt1/assignment_loss‚1.339839961546074íÜÃ_ÓÙ trainer/global_step‚6449 1loss/SRqqt1/detection_loss‚1.7835732246083873 &loss/total_loss‚41.378681064787656 1loss/FBt4/assignment_loss‚0.12053801095922526 /loss/FRt4/detection_loss‚1.5014493653208625 epoch‚5 /loss/FRt1/assignment_loss‚3.127511907455533 2loss/SRqqt2/assignment_loss‚1.7922347411513329 2loss/SRqqt4/assignment_loss‚0.7260107439132812 1loss/FBt2/assignment_loss‚0.29869432218583825 /loss/SRqqt2/detection_loss‚2.41551753713025 _step‚530 2loss/SRqqt3/assignment_loss‚1.4985260561640774 1loss/FBt3/assignment_loss‚0.32030188851058483 .loss/FRt2/detection_loss‚2.819441228201895 /loss/FBt3/detection_loss‚2.4050681442022324 /loss/FBt4/detection_loss‚0.7616167866280708 1loss/SRqqt1/assignment_loss‚1.339839961546074 1loss/FBt1/assignment_loss‚0.20413777008652687 % _timestamp‚1.7884215938574188e+09 _runtime‚5093.594892879 /loss/FRt4/assignment_loss‚1.612547021739337 1loss/SRqqt3/detection_loss‚1.4537898128586155 0loss/FRt2/assignment_loss‚3.0464604551141914 0loss/FRt3/assignment_loss‚2.8032632280479777 /loss/FRt1/detection_loss‚2.3837658922339595 -loss/FRt3/detection_loss‚2.22051500100078 0loss/SRqqt4/detection_loss‚1.091165843271259 /loss/FBt1/detection_loss‚1.8821014493703843 .loss/FBt2/detection_loss‚3.770610673086984’¡Ž <èïÔ‚(:’ àËäÔÔÀ~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.284' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.656799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðÞ»tÕ‚(:é àËäÔèºæ€ disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0$ memory_percent‚90.65747261047363* proc.memory.percent‚16.921329498291016 network.sent‚ 1.4913046e+07( disk.nvme0c0n1.out‚22093.25634765625 disk.drbd2265.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚22093.25634765625 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 cpu‚25.34328930124964 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.65625 network.recv‚ 1.908122e+06* disk./.usagePercent‚17.359577256192352 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 326.2265625 disk.nvme0c0n1.in‚4.65625 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.65625$ disk./.usageGB‚258.6702461242676 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 326.2265625& disk.nvme0n1.out‚22093.25634765625! proc.memory.rssMB‚ 2772.390625 disk.sda3.out‚ 326.2265625 disk.rbd1.out‚ 2.759765625 disk.drbd1716.in‚0( proc.memory.availableMB‚ 1530.6796875`† ÈÖ‚(:’ ïËäÔíú¨~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^9£,xׂ(:í ïËäÔω¾€% disk./.usageGB‚258.68282318115234 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 326.76171875 disk.sda.out‚ 326.76171875 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.65625 disk.sda.in‚ 126.77734375$ memory_percent‚90.65768718719482 cpu‚25.34243682935319 disk.drbd2265.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17) disk./.usagePercent‚17.36042131303274 disk.nvme0n1.in‚4.65625& disk.nvme0n1.out‚22141.43603515625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚22141.43603515625* proc.memory.percent‚16.921329498291016 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 326.76171875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.65625) proc.memory.availableMB‚ 1530.64453125 network.sent‚ 1.5021211e+07# disk.dm-0.out‚22141.43603515625 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 network.recv‚ 1.913169e+06+¬r’Ø‚(:‡ þËäÔÞº˜~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆº›xÙ‚(:í þËäÔÍ¡©€ network.recv‚ 1.917601e+06$ memory_percent‚90.65923690795898* proc.memory.percent‚16.921329498291016 disk.rbd1.out‚ 2.759765625' proc.memory.availableMB‚ 1530.390625! proc.memory.rssMB‚ 2772.390625 cpu‚25.341539321811375 proc.cpu.threads‚17 disk.rbd1.in‚0( disk.nvme0c0n1.out‚22197.11962890625 network.sent‚ 1.5027246e+07 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚22197.11962890625 disk.drbd2266.out‚0 disk.sda.out‚ 327.46484375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.65625 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0% disk./.usageGB‚258.67938232421875 disk.sda1.out‚0 disk.sda3.out‚ 327.46484375* disk./.usagePercent‚17.360190394236856 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 327.46484375 disk.drbd1716.in‚0& disk.nvme0n1.out‚22197.11962890625 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.65625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.65625R ŒT2Ú- _wandbruntime‚5131  _runtime‚5131÷¯ÇÛÁ & lr-AdamW/pg1‚8.800710114276189e-05 & lr-AdamW/pg2‚8.800710114276189e-05 trainer/global_step‚6499 % _timestamp‚1.7884216315332606e+09 _runtime‚5131.270781972 _step‚531-ðØõÌÜÆ trainer/global_step‚6499 % _timestamp‚1.7884216315332606e+09 _runtime‚5131.270781972 _step‚531 & lr-AdamW/pg1‚8.800710114276189e-05 & lr-AdamW/pg2‚8.800710114276189e-05“©ËæN2Ý- _wandbruntime‚5131  _runtime‚5131dà¬cá'ÞÛO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94515486 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31104343 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122690989 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚383356784123904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚532 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412681282241024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5710043480576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368010083306496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚5131.272038694 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚6499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443003691520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚532 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377913124069888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377913124069888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31103877 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412681282241024 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5710219687424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5710043480576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚383356109758464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5443660054016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5423618969554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412681306181120 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122711027 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373433030798934 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86250337 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373433702276050 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚368010083306496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚383356784123904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚383356109758464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚5131.272038694 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377913106066944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94515488 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377913106066944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86250805 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94515488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚418391525868544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122690989 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚6499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5443003691520 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025”A»t2à- _wandbruntime‚5132  _runtime‚51321çO~áäM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚533 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377969459105792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122729507 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚6499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383413956603904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368064970397696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5444479495168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5444479495168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5711076189184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217259011 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 PBDeviceStatsMonitor.on_trE=²Nl ain_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788421632267967e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377969477108736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217259011 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217238945 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412742956918784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217259011 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 217238945 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788421632267967e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚373488751364458 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377969477108736 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94529504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31108523 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚533 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377969459105792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5443823119360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚383413282225152 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122729507 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚6499 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55154630 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚383413956603904 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55154628 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5443823119360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368064987437056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86263151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418453853945856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚368064970397696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5444479495168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31108971 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5424435404726 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430•é6">2ã- _wandbruntime‚5132  _runtime‚5132åóãYäÓ 2loss/SRqqt4/assignment_loss‚1.0539089044183492 1loss/FBt4/assignment_loss‚0.15137386540136838 .loss/FBt2/detection_loss‚5.466386377811432 /loss/FBt4/detection_loss‚0.5222430392871805 _runtime‚5132.006971295 /loss/FRt1/assignment_loss‚3.663678530189726 /loss/FRt3/assignment_loss‚2.745937164752714 0loss/FRt4/assignment_loss‚1.5079829319048736 1loss/FBt1/assignment_loss‚0.37590407285218436 %loss/total_loss‚47.53204188072465 epoch‚5 1loss/SRqqt2/assignment_loss‚2.376852259039879 0loss/SRqqt3/assignment_loss‚1.57466405414437 1loss/FBt3/assignment_loss‚0.15130212996155024 /loss/FRt1/detection_loss‚2.5936972283654742 1loss/SRqqt4/detection_loss‚0.9991172279417515 % _timestamp‚1.7884216322694793e+09 /loss/FRt4/detection_loss‚1.5739355238817505 0loss/SRqqt1/detection_loss‚1.980594434908458 1loss/SRqqt2/detection_loss‚2.5252655849737278 trainer/global_step‚6499 0loss/FRt2/assignment_loss‚3.7714019656181335 2loss/SRqqt1/assignment_loss‚1.5401944058890245 /loss/FRt2/detection_loss‚3.0631410519282025 1loss/SRqqt3/detection_loss‚1.7407266387813969 /loss/FBt3/detection_loss‚2.6596154967943826 0loss/FBt2/assignment_loss‚0.4835461676120758 _step‚534 /loss/FRt3/detection_loss‚1.9429170373947389 -loss/FBt1/detection_loss‚3.06765578687191t‚ãÓ^åØ 2loss/SRqqt4/assignment_loss‚1.0539089044183492 1loss/FBt4/assignment_loss‚0.15137386540136838 .loss/FBt2/detection_loss‚5.466386377811432 /loss/FBt4/detection_loss‚0.5222430392871805 _runtime‚5132.006971295 /loss/FRt1/assignment_loss‚3.663678530189726 /loss/FRt3/assignment_loss‚2.745937164752714 0loss/FRt4/assignment_loss‚1.5079829319048736 1loss/FBt1/assignment_loss‚0.37590407285218436 %loss/total_loss‚47.53204188072465 epoch‚5 1loss/SRqqt2/assignment_loss‚2.376852259039879 0loss/SRqqt3/assignment_loss‚1.57466405414437 1loss/FBt3/assignment_loss‚0.15130212996155024 /loss/FRt1/detection_loss‚2.5936972283654742 1loss/SRqqt4/detection_loss‚0.9991172279417515 % _timestamp‚1.7884216322694793e+09 /loss/FRt4/detection_loss‚1.5739355238817505 0loss/SRqqt1/detection_loss‚1.980594434908458 1loss/SRqqt2/detection_loss‚2.5252655849737278 trainer/global_step‚6499 0loss/FRt2/assignment_loss‚3.7714019656181335 2loss/SRqqt1/assignment_loss‚1.5401944058890245 /loss/FRt2/detection_loss‚3.0631410519282025 1loss/SRqqt3/detection_loss‚1.7407266387813969 /loss/FBt3/detection_loss‚2.6596154967943826 0loss/FBt2/assignment_loss‚0.4835461676120758 _step‚534 /loss/FRt3/detection_loss‚1.9429170373947389 -loss/FBt1/detection_loss‚3.06765578687191–D/’æ‚(:‡ ÌäÔ¾úµ~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶c–>jç‚(:ß ÌäÔÑú¡ disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 2.759765625 network.sent‚ 1.5033125e+07 disk.rbd0.out‚0' disk.nvme0c0n1.out‚22246.5810546875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 network.recv‚ 1.921996e+06* disk./.usagePercent‚17.360969681171085 disk.dm-0.in‚4.65625 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.65625 disk.sda3.out‚ 328.09375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.out‚ 328.09375! proc.memory.rssMB‚ 2772.390625* proc.memory.percent‚16.921329498291016 cpu‚25.34073238192155$ disk./.usageGB‚258.6909942626953 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚90.65790176391602 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.65625 disk.sda2.out‚0" disk.dm-0.out‚22246.5810546875 disk.dm-1.out‚ 328.09375 disk.drbd2266.in‚0% disk.nvme0n1.out‚22246.5810546875' proc.memory.availableMB‚ 1530.609375 proc.cpu.threads‚17.9?“è‚(:ˆ œÌäÔ©°§€ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶H‹Ýmé‚(:â œÌäÔýˆ”* disk./.usagePercent‚17.358601867486893 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17$ disk.nvme0n1.out‚22332.287109375 disk.sda2.in‚0) proc.memory.availableMB‚ 1530.85546875 disk.sda1.in‚0 disk.nvme0n1.in‚4.65625 disk.dm-1.out‚ 328.609375! proc.memory.rssMB‚ 2772.390625 cpu‚25.339844325510438 network.sent‚ 1.5142268e+07 network.recv‚ 1.92881e+06 disk.nvme0c0n1.in‚4.65625& disk.nvme0c0n1.out‚22332.287109375 disk.sda3.out‚ 328.609375 disk.dm-1.in‚ 126.77734375$ memory_percent‚90.65639972686768 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0% disk./.usageGB‚258.65571212768555 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 2.759765625 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.65625 disk.drbd2266.out‚0 disk.sda.out‚ 328.609375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0! disk.dm-0.out‚22332.287109375tEiE2ê- _wandbruntime‚5169  _runtime‚5169BÐqÁÇëÁ trainer/global_step‚6549 % _timestamp‚1.7884216700339901e+09 _runtime‚5169.771501502 _step‚535 & lr-AdamW/pg1‚8.679031925808342e-05 & lr-AdamW/pg2‚8.679031925808342e-05§3Á–ÌìÆ & lr-AdamW/pg1‚8.679031925808342e-05 & lr-AdamW/pg2‚8.679031925808342e-05 trainer/global_step‚6549 % _timestamp‚1.7884216700339901e+09 _runtime‚5169.771501502 _step‚535—„_2í- _wandbruntime‚5169  _runtime‚5169L“Ñá'îÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380730758009344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218853367 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 trainer/global_step‚6549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370755289834496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚5169.772444001 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5483989497344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380730776012288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376219744143458 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123637079 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123617041 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55554862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421516783155200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55554860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414·õpà ïàO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚370755289834496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 _runtime‚5169.772444001 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚386215421872128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421516979107840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415763819124224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5483989497344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5483989497344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218833327 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370755272795136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_ps`gool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380730776012288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚386214747506688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380730758009344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95216286 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386215421872128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31330611 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚376219744143458 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123637079 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86885473 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752964030976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123617041 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55554862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421516783155200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5484645859840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55554860 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5463799871206 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386214747506688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86885034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123617041 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380730758009344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218853367 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 trainer/global_step‚6549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5484645859840˜ÿKQ¾2ð- _wandbruntime‚5170  _runtime‚5170šˆ²àë&ñåM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5465270744134 _runtime‚5170.747501369 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚6549 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123635495 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380787129051136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5485465300992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55563028 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884216710095067e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86898079 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386272594352128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376275464709190 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5464616306170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5485465300992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55563030 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421579298174976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123635495 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415825469861888 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123655559 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415825469861888 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218885863 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421579485725696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95230304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31334616 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55563028 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86897644 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95230302 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 % _timestamp‚1.7884216710095067e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123655559 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386272594352128 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370810193965056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86898079 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218885863 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386272594352128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376275464709190 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5464616306170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5485465300992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55563030 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421579298174976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123635495 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5484808925184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95230304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 % _timestamp‚1.7884217093603616e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383548427954688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 _step‚540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373500496362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5505289648370 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424642275730944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5525631665664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418846356007424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5209.098137154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389073385254912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844p,ÿæ'àO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383548427954688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚373500479323136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418846356007424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5209.098137154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5524975303168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389073385254912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124563131 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55963262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383548409951744 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383548409951744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87521910 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31559101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5795919723520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124543093 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389073385254912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5504635210614 % _timestamp‚1.7884217093603616e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95917086 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220460179 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5525631665664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5524975303168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5796079153152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383548427954688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87522363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424642459100672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚379005786010866 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124543093 _step‚540 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389074059620352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚373500496362496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31558650 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚418846379947520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5505289648370 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424642275730944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389074059620352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95917086 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5525631665664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220480219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379005114533750 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95917088œ«Ú *2‚- _wandbruntime‚5209  _runtime‚5209g¸*¬ë&ƒåM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5209.826129007 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418908006745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220492649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5526451106816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚6599 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5525794731008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389131232100352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5506106083542 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95931104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55971428 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884217100880215e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0µr&„êM ;5DeviceStatsMonitor.on_train_žÞÊ&batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124561547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚424704972434944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124581611 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5526451106816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87534624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚6599 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220512715 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389131232100352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5525794731008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389131232100352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5526451106816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95931102 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383604762990592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383604780993536 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95931104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95931102 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220512715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5506106083542 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95931104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55971428 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884217100880215e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55971430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5209.826129007 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚418908030685184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚541 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124581611 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚424704782787072 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418908006745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5525794731008 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220492649 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125469145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚427767763441664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376245702890496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125489183 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386366061894144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 $ _timestamp‚1.788421748697804e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56371660 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546124987778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96617888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96617886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391932697368576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421928892890624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381791827878274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125469145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31785760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545470550022 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5839034174976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ÄšNµö‘ßO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545470550022 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5839034174976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5566617471488 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222087031 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5565961108992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421928916830720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386366061894144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386366079897088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31786196 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚5248.435631679 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88157420 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96617888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96617886 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚376245685851136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386366079897088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125469145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚427767763441664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5566617471488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125489183 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376245702890496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚391932023003136 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125489183 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386366061894144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56371662 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 $ _timestamp‚1.788421748697804e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.xûïfreed‚56371660 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚391932697368576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546124987778 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88157858 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381791156401158 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96617888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96617886 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚391932697368576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚391932023003136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421928892890624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381791827878274 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125469145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31785760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 iÇŠÄ2’- _wandbruntime‚5249  _runtime‚5249=&B2é&“ãM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚ 5249.17651259 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31790238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96631902 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391989869848576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430óãË‘î&”èM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚427830278877184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125487599 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222119501 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚386422432935936 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376300589981696 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125487599 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125507663 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚ 5249.17651259 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125507663 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚386422414932992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88170066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31790238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5546941422950 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96631904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222139567 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222119501 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96631902 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5839735248896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚386422432935936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5567436912640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421990543628288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚391989869848576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56379830 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96631902 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5566780536832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31790661 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128¡’àÞ2•- _wandbruntime‚5249  _runtime‚5249<32Z–Ô 1loss/SRqqt1/assignment_loss‚1.519376214392568 &loss/total_loss‚42.825615611566334 _step‚546 /loss/FRt1/detection_loss‚1.9370923736628065 .loss/FRt2/detection_loss‚2.624099151127868 0loss/FRt1/assignment_loss‚2.9535756402827325 0loss/FBt2/assignment_loss‚0.5162062265444547 1loss/SRqqt1/detection_loss‚1.7971540435471318 epoch‚5 trainer/global_step‚6649 1loss/FBt4/assignment_loss‚0.15056714072345498 /loss/FBt3/detection_loss‚2.7231793950001397 2loss/SRqqt3/assignment_loss‚1.7584217265248299 /loss/FRt4/detection_loss‚2.0467420369386673 0loss/SRqqt2/detection_loss‚2.626851533398484 .loss/FBt2/detection_loss‚2.661023815472921 _runtime‚5249.177625089 1loss/FBt1/assignment_loss‚0.21293535036966205 .loss/FRt3/detection_loss‚2.393531598670538 1loss/SRqqt3/detection_loss‚1.8552600212947086 $ _timestamp‚1.788421749440147e+09 /loss/FRt2/assignment_loss‚3.905224449104733 /loss/FRt3/assignment_loss‚2.807425455991612 /loss/FRt4/assignment_loss‚1.614009469054466 2loss/SRqqt4/assignment_loss‚0.9484543786781455 1loss/FBt3/assignment_loss‚0.26622588908260997 1loss/SRqqt4/detection_loss‚1.0917347015014716 /loss/FBt1/detection_loss‚1.9971723301070077 /loss/FBt4/detection_loss‚0.6881537622937202 2loss/SRqqt2/assignment_loss‚1.7311989078015992ˆ"Ê¿_—Ù /loss/FRt4/assignment_loss‚1.614009469054466 2loss/SRqqt4/assignment_loss‚0.9484543786781455 1loss/FBt3/assignment_loss‚0.26622588908260997 1loss/SRqqt4/detection_loss‚1.0917347015014716 /loss/FBt1/detection_loss‚1.9971723301070077 /loss/FBt4/detection_loss‚0.6881537622937202 2loss/SRqqt2/assignment_loss‚1.7311989078015992 1loss/SRqqt1/assignment_loss‚1.519376214392568 &loss/total_loss‚42.825615611566334 _step‚546 /loss/FRt1/detection_loss‚1.9370923736628065 .loss/FRt2/detection_loss‚2.624099151127868 0loss/FRt1/assignment_loss‚2.9535756402827325 0loss/FBt2/assignment_loss‚0.5162062265444547 1loss/SRqqt1/detection_loss‚1.7971540435471318 epoch‚5 trainer/global_step‚6649 1loss/FBt4/assignment_loss‚0.15056714072345498 /loss/FBt3/detection_loss‚2.7231793950001397 2loss/SRqqt3/assignment_loss‚1.7584217265248299 /loss/FRt4/detection_loss‚2.0467420369386673 0loss/SRqqt2/detection_loss‚2.626851533398484 .loss/FBt2/detection_loss‚2.661023815472921 _runtime‚5249.177625089 1loss/FBt1/assignment_loss‚0.21293535036966205 .loss/FRt3/detection_loss‚2.393531598670538 1loss/SRqqt3/detection_loss‚1.8552600212947086 $ _timestamp‚1.788421749440147e+09 /loss/FRt2/assignment_loss‚3.905224449104733 /loss/FRt3/assignment_loss‚2.807425455991612¢žI’˜‚(:‡ öÌäÔ梔~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pËoäy™‚(:î öÌäÔÒó‡ disk.nvme0n1.in‚4.65625* proc.memory.percent‚16.921329498291016 disk.drbd2266.out‚0 disk.dm-0.in‚4.65625 network.sent‚ 1.5382851e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.65625 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚22618.30029296875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0& disk.nvme0n1.out‚22618.30029296875( disk.nvme0c0n1.out‚22618.30029296875 disk.sda3.out‚ 331.66015625 disk.drbd2080.in‚0 network.recv‚ 1.957841e+06* disk./.usagePercent‚17.362367994323105) proc.memory.availableMB‚ 1530.11328125! proc.memory.rssMB‚ 2772.390625 cpu‚25.334853897993572 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 331.66015625# memory_percent‚90.6609296798706% disk./.usageGB‚258.71183013916016 disk.sda.out‚ 331.66015625 disk.drbd1716.in‚0 disk.drbd1716.out‚0£³ì’š‚(:‡ …ÍäÔщª~ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!å u›‚(:ê …ÍäÔÊùÄ€ cpu‚25.334051421132152 disk.rbd1.out‚ 2.759765625 disk.sda.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 1.964118e+06% disk./.usageGB‚258.71657943725586 disk.sda.out‚ 332.16796875 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.65625 disk.sda2.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚22661.28076171875 network.sent‚ 1.5491521e+07 disk.sda3.in‚ 126.77734375$ proc.memory.availableMB‚1530.125! proc.memory.rssMB‚ 2772.390625* proc.memory.percent‚16.921329498291016( disk.nvme0c0n1.out‚22661.28076171875 disk.dm-1.out‚ 332.16796875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0& disk.nvme0n1.out‚22661.28076171875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.65625 disk.sda3.out‚ 332.16796875 disk.sda1.in‚0 disk.sda2.out‚0* disk./.usagePercent‚17.362686723703234 disk.drbd2265.in‚0$ memory_percent‚90.66085815429688 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.65625~,¥Z’œ‚(:‡ ”ÍäÔ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚165.697' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•\ú•u‚(:ê ”ÍäÔõ²Ù€ disk.drbd2080.in‚0* disk./.usagePercent‚17.363579677982663 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 332.55078125 disk.sda2.out‚0( proc.memory.availableMB‚ 1529.8828125 disk.rbd1.out‚ 2.759765625 disk.drbd1716.in‚0 disk.dm-1.out‚ 332.55078125 cpu‚25.333246830645983 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2772.390625 disk.sda3.out‚ 332.55078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚4.65625" disk.dm-0.out‚22710.1884765625# memory_percent‚90.6623363494873* proc.memory.percent‚16.921329498291016 network.sent‚ 1.5497821e+07% disk./.usageGB‚258.72988510131836 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.recv‚ 1.969298e+06 disk.sda1.out‚0 disk.nvme0n1.in‚4.65625% disk.nvme0n1.out‚22710.1884765625 disk.nvme0c0n1.in‚4.65625' disk.nvme0c0n1.out‚22710.1884765625 disk.sda3.in‚ 126.77734375Y8`P2ž- _wandbruntime‚5287  _runtime‚5287{†l1ÇŸÁ & lr-AdamW/pg1‚8.313792865799896e-05 & lr-AdamW/pg2‚8.313792865799896e-05 trainer/global_step‚6699 % _timestamp‚1.7884217875001664e+09 _runtime‚5287.237727657 _step‚547Á"(Ì Æ & lr-AdamW/pg1‚8.313792865799896e-05 & lr-AdamW/pg2‚8.313792865799896e-05 trainer/global_step‚6699 % _timestamp‚1.7884217875001664e+09 _runtime‚5287.237727657 _step‚547£zö\™2¡- _wandbruntime‚5287  _runtime‚5287¡Æ*ñ¢ÛO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5607603277312 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/re×8€ÓR%served_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚394791335116800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5606946914816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394791335116800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394790660751360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126415235 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223733923 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378990892379136 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126395197 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97318688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88792364 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5882093979136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126395197 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268566 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97318686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389183731839488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚430893547693056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚430893355934720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5586305889430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5586960327186 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚389183731839488 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126395197 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚384577198268566 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97318686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32012304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5606946914816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97318686 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394790660751360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389183731839488 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚6699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5587122324394 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5608422718464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379045796509696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384632918834090 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5882790197248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5287.964783928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88805514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394848507596800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488e¼¤éˆ ¦êM EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88805064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5607766342656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56788230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32017284 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223746353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394848507596800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚389240066875392 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97332702 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425073104451584 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425073080511488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5882951710720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126433715 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56788228 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223746353 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytZM±hes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚6699 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223766419 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97332702 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5587122324394 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5608422718464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379045796509696 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32016836 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389240084878336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97332704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚384632918834090 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5587776762358 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5608422718464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379045813549056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126413651 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5607766342656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394847833218048 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389240066875392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5882790197248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚5287.964783928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97332704 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223766419 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚384633590311414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88805514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98019486 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889ôFŠÒnµàO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127321249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387363240135974 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225340735 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32239696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428093966657024 % _timestamp‚1.7884218269094613e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32239254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397649298499584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89428158 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5627795666594 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127321249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98019488 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5648589083136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392001365778944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57188462 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397649298499584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚397649972865024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392001383781888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5647932720640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚381736098907136 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127341287 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428093990597120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98019486 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚434081410150400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884218276510613e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98033502 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225373205 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397707145345024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3874189607014987c†ï&¸éM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392057736820736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428155617394688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32243750 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚434081410150400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127339703 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89440378 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428155641334784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225393271 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚6749 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5925954269184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392057718817792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884218276510613e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98033502 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392057718817792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225393271 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5648752148480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98033504 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98033502 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127359767 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚381791020077056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5628612101766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225373205 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127339703 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397707145345024 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98033504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387418960701498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57196630 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚434081595603968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2©o©72¹- _wandbruntime‚5327  _runtime‚5327)´PèXºÒ /loss/FRt2/detection_loss‚3.1577503319297517 2loss/SRqqt1/assignment_loss‚1.3499557215607527 2loss/SRqqt3/assignment_loss‚1.6323172105175834 _step‚554 /loss/FRt1/detection_loss‚2.1412758882398957 .loss/FBt2/detection_loss‚4.452640622854233 epoch‚5 0loss/FRt4/assignment_loss‚1.4085457745720358 1loss/FBt1/assignment_loss‚0.41177756851539016 0loss/FBt2/assignment_loss‚0.2835867945832433 .loss/FRt4/detection_loss‚1.746003157660073 0loss/SRqqt2/detection_loss‚2.689025318080729 $ _timestamp‚1.788421827653146e+09 2loss/SRqqt4/assignment_loss‚1.1937981671584186 0loss/SRqqt4/detection_loss‚1.049255836087821 1loss/FBt4/assignment_loss‚0.14479458559569433 0loss/FRt1/assignment_loss‚2.9914347926775613 0loss/FRt2/assignment_loss‚3.6032090059348514 0loss/FRt3/assignment_loss‚3.0253100683291754 /loss/FBt1/detection_loss‚2.0346637964248657 $loss/total_loss‚44.3699860835516 _runtime‚ 5327.39059125 1loss/FBt3/assignment_loss‚0.11155569204129279 .loss/FRt3/detection_loss‚2.232620665927728 1loss/SRqqt1/detection_loss‚2.2054485353556545 /loss/FBt4/detection_loss‚0.5907561936126113 .loss/FBt3/detection_loss‚2.397094115614891 1loss/SRqqt2/assignment_loss‚1.842977368018844 1loss/SRqqt3/detection_loss‚1.6741888722585094 trainer/global_step‚6749šdä;]»× .loss/FBt3/detection_loss‚2.397094115614891 1loss/SRqqt2/assignment_loss‚1.842977368018844 1loss/SRqqt3/detection_loss‚1.6741888722585094 trainer/global_step‚6749 /loss/FRt2/detection_loss‚3.1577503319297517 2loss/SRqqt1/assignment_loss‚1.3499557215607527 2loss/SRqqt3/assignment_loss‚1.6323172105175834 _step‚554 /loss/FRt1/detection_loss‚2.1412758882398957 .loss/FBt2/detection_loss‚4.452640622854233 epoch‚5 0loss/FRt4/assignment_loss‚1.4085457745720358 1loss/FBt1/assignment_loss‚0.41177756851539016 0loss/FBt2/assignment_loss‚0.2835867945832433 .loss/FRt4/detection_loss‚1.746003157660073 0loss/SRqqt2/detection_loss‚2.689025318080729 $ _timestamp‚1.788421827653146e+09 2loss/SRqqt4/assignment_loss‚1.1937981671584186 0loss/SRqqt4/detection_loss‚1.049255836087821 1loss/FBt4/assignment_loss‚0.14479458559569433 0loss/FRt1/assignment_loss‚2.9914347926775613 0loss/FRt2/assignment_loss‚3.6032090059348514 0loss/FRt3/assignment_loss‚3.0253100683291754 /loss/FBt1/detection_loss‚2.0346637964248657 $loss/total_loss‚44.3699860835516 _runtime‚ 5327.39059125 1loss/FBt3/assignment_loss‚0.11155569204129279 .loss/FRt3/detection_loss‚2.232620665927728 1loss/SRqqt1/detection_loss‚2.2054485353556545 /loss/FBt4/detection_loss‚0.5907561936126113ªÞÎ}¿¼‚(:’ ÐÍäÔ¿Œ…~ gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.269' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.653800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01±ô²h½‚(:Ý ÐÍäÔÍŸ¦€ network.recv‚ 1.989134e+06 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 334.57421875 disk.sda2.out‚0$ memory_percent‚90.66147804260254 network.sent‚ 1.5624296e+07 disk.rbd1.out‚ 2.759765625 cpu‚25.3300371729087 disk.dm-0.in‚4.65625 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.65625! disk.nvme0n1.out‚ 22897.796875 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0# disk.nvme0c0n1.out‚ 22897.796875( proc.memory.availableMB‚ 1530.0234375 disk.dm-1.out‚ 334.57421875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.65625! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17* disk./.usagePercent‚17.363888679076492 disk.sda3.in‚ 126.77734375 disk.dm-0.out‚ 22897.796875 disk.sda.out‚ 334.57421875 disk.sda2.in‚0 disk.rbd0.out‚0% disk./.usageGB‚258.73448944091797 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375* proc.memory.percent‚16.921329498291016oÏ ¾‚(:… ßÍäÔÇ“~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝG•e¿‚(:Ú ßÍäÔü°Ý€% disk.nvme0c0n1.out‚22943.51953125 disk.sda3.out‚ 335.03125* disk./.usagePercent‚17.361206488140258 disk.drbd2266.out‚0 cpu‚25.329252294117342# disk./.usageGB‚258.694522857666 disk.dm-0.in‚4.65625 disk.dm-1.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.759765625 disk.nvme0n1.in‚4.65625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.65625 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1530.02734375 network.recv‚ 1.994633e+06 disk.sda1.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 disk.dm-0.out‚22943.51953125 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0$ memory_percent‚90.66145420074463 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 335.03125 disk.rbd0.in‚0! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.573231e+07 disk.dm-1.out‚ 335.03125 disk.drbd1716.in‚0# disk.nvme0n1.out‚22943.51953125Œ$Œ2À- _wandbruntime‚5365  _runtime‚5365 é°ÃÇÁÁ & lr-AdamW/pg2‚8.070383463783304e-05 trainer/global_step‚6799 % _timestamp‚1.7884218659980288e+09 _runtime‚5365.735543951 _step‚555 & lr-AdamW/pg1‚8.070383463783304e-05pT¬}ÌÂÆ & lr-AdamW/pg2‚8.070383463783304e-05 trainer/global_step‚6799 % _timestamp‚1.7884218659980288e+09 _runtime‚5365.735543951 _step‚555 & lr-AdamW/pg1‚8.070383463783304e-05«øS™Æ2Ã- _wandbruntime‚5365  _runtime‚5365&[ˆL5ÄÛO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5968244968448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5688918526464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884218659988322e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431176527480320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.larWÅ߬ge_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚5365.736774724 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384481305435136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437144772448768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667976568246 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5689574888960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128267339 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5968087635968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390149953480498 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226967587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90063116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5688918526464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98720288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400507936247808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚556 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384481322474496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872©w5æ'ÅàO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394819035724288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚431176527480320 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚400508610613248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚5365.736774724 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384481305435136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32466256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚442 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226987627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437144772448768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚390149282003382 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667976568246 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5689574888960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394819035724288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128267339 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚431176503540224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689574888960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5968087635968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚390149953480498 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226967587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90063116 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5688918526464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98720288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚400507936247808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚556 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384481322474496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚400507936247808 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437144591176192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5968244968448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5688918526464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884218659988322e+09¬Ö’/2Æ- _wandbruntime‚5366  _runtime‚5366¹ ”ôë&ÇåM trainer/global_step‚6799 % _timestamp‚1.7884218667340288e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400565108714496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128285819 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437207289677312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 _runtime‚5366.472080274 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚557 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414ÅbD.ÈêM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57605028 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90075835 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5690394330112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437207289677312 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5669447441174 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400565783093248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98734302 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚437 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32471242 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚390205002568906 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227000057 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98734304 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227000057 _runtime‚5366.472080274 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128265755 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚431238154277888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚435 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚557 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437207104223744 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227020123 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5968949945856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚6799 % _timestamp‚1.7884218667340288e+09 XIDeviceStatsMonitor.on_train_batch_end/reserveô‡úÕÂd_bytes.large_pool.allocated‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400565108714496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5689737954304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98734304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690394330112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128285819 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚390205674046230 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384536226605056 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5969111459328 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227020123 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5689737954304 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128265755 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394875370760192 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32694302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5405.160034722 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58005260 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440270329460224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5708811907654 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6011095181312 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32694754 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚397636669663744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403366573996032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5729904332288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129193391 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228614479 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440270135604736 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5708811907654 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403367248361472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6011095181312 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99421088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403367248361472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5729904332288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392935995347906 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚387226529002496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32694302 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6011265096704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚397636669663744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434259064363520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚5405.160034722 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129173353 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5730560694784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58005260 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5709466345410 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90699562 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚440270329460224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99435102 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387281433133056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387281416093696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5731380135936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884219061638165e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5730723760128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5709628342618 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440332666146304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99435104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392991715913638 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5730723760128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6849 _runtime‚5405.902094394û¢úµð&ÚêM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5731380135936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434320691161088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚397693022702592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228646975 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90712172 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884219061638165e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5730723760128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434320715101184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5709628342618 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403424420841472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99435102 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228626909 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440332843211264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440332666146304 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99435104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58013430 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚392991715913638 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5730723760128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129211871 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6849 _runtime‚5405.902094394 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚403424420841472 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99435102 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚387281433133056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5710282780582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚387281416093696 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129191807 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430±h0?2Û- _wandbruntime‚5405  _runtime‚5405‰JÛÀZÜÔ /loss/FRt1/detection_loss‚2.1226297237656335 /loss/FRt4/detection_loss‚1.6215035488893246 /loss/FBt1/detection_loss‚2.6182814478874206 1loss/SRqqt1/detection_loss‚2.0179746492606836 1loss/SRqqt2/detection_loss‚2.5287299237468024 .loss/FBt2/detection_loss‚2.154958701796002 _step‚562 2loss/SRqqt2/assignment_loss‚1.7823036417195743 0loss/FBt1/assignment_loss‚0.5210202940739691 0loss/FBt2/assignment_loss‚0.5069133446862301 1loss/SRqqt4/detection_loss‚0.9317550809681415 trainer/global_step‚6849 % _timestamp‚1.7884219061656053e+09 /loss/FRt1/assignment_loss‚3.668663597468174 2loss/SRqqt4/assignment_loss‚0.8379478753140817 /loss/FBt3/detection_loss‚2.9958197832107545 2loss/SRqqt3/assignment_loss‚1.3024135186987102 0loss/FBt4/assignment_loss‚0.1872553492867107 0loss/FRt2/assignment_loss‚3.5699324448903402 /loss/FRt3/assignment_loss‚3.220138130234737 &loss/total_loss‚43.429335655320756 epoch‚5 _runtime‚5405.903082379 /loss/FRt2/detection_loss‚2.7307163874308267 0loss/FRt4/assignment_loss‚1.2706090304358253 0loss/FBt3/assignment_loss‚0.3228310185484588 0loss/SRqqt3/detection_loss‚1.750407725010278 /loss/FBt4/detection_loss‚0.7451723114792983 1loss/SRqqt1/assignment_loss‚1.778201699256897 .loss/FRt3/detection_loss‚2.243156427261876' _ÝÙ _step‚562 2loss/SRqqt2/assignment_loss‚1.7823036417195743 0loss/FBt1/assignment_loss‚0.5210202940739691 0loss/FBt2/assignment_loss‚0.5069133446862301 1loss/SRqqt4/detection_loss‚0.9317550809681415 trainer/global_step‚6849 % _timestamp‚1.7884219061656053e+09 /loss/FRt1/assignment_loss‚3.668663597468174 2loss/SRqqt4/assignment_loss‚0.8379478753140817 /loss/FBt3/detection_loss‚2.9958197832107545 2loss/SRqqt3/assignment_loss‚1.3024135186987102 0loss/FBt4/assignment_loss‚0.1872553492867107 0loss/FRt2/assignment_loss‚3.5699324448903402 /loss/FRt3/assignment_loss‚3.220138130234737 &loss/total_loss‚43.429335655320756 epoch‚5 _runtime‚5405.903082379 /loss/FRt2/detection_loss‚2.7307163874308267 0loss/FRt4/assignment_loss‚1.2706090304358253 0loss/FBt3/assignment_loss‚0.3228310185484588 0loss/SRqqt3/detection_loss‚1.750407725010278 /loss/FBt4/detection_loss‚0.7451723114792983 1loss/SRqqt1/assignment_loss‚1.778201699256897 .loss/FRt3/detection_loss‚2.243156427261876 /loss/FRt1/detection_loss‚2.1226297237656335 /loss/FRt4/detection_loss‚1.6215035488893246 /loss/FBt1/detection_loss‚2.6182814478874206 1loss/SRqqt1/detection_loss‚2.0179746492606836 1loss/SRqqt2/detection_loss‚2.5287299237468024 .loss/FBt2/detection_loss‚2.154958701796002²0EC\’Þ‚(:‡ ›ÎäÔý‹ ~ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚165.697' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.1394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DGEì{ß‚(:ð ›ÎäÔìíº€ disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚23145.60986328125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.671875 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375% disk./.usageGB‚258.75135803222656 disk.drbd1716.in‚0 disk.sda2.out‚0# memory_percent‚90.6620740890503& disk.nvme0n1.out‚23145.60986328125 disk.sda3.out‚ 337.27734375 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 cpu‚25.32612965274908 network.sent‚ 1.5857693e+07( disk.nvme0c0n1.out‚23145.60986328125 disk.drbd2265.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1529.92578125! proc.memory.rssMB‚ 2772.390625* disk./.usagePercent‚17.365020744392904 disk.rbd0.out‚0 disk.dm-1.out‚ 337.27734375 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625* proc.memory.percent‚16.921329498291016 network.recv‚ 2.012533e+06 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.sda.out‚ 337.27734375!ÛÉ…’à‚(:‡ ªÎäÔ÷¹™~ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏãU`á‚(:Õ ªÎäÔ´áš‚ network.recv‚ 2.019263e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1529.92578125 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 337.67578125 disk.sda1.out‚0 disk.nvme0c0n1.out‚23204.75# memory_percent‚90.6620740890503 cpu‚25.32535362916627% disk./.usageGB‚258.78124618530273 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.sda2.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921329498291016* disk./.usagePercent‚17.367026563423675 disk.dm-0.out‚23204.75 disk.rbd0.out‚0 network.sent‚ 1.5966615e+07 disk.rbd1.out‚ 2.759765625 disk.sda3.out‚ 337.67578125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 disk.dm-1.out‚ 337.67578125 disk.nvme0n1.out‚23204.75 disk.drbd2080.in‚0 proc.cpu.threads‚17º‚Ë2â- _wandbruntime‚5444  _runtime‚5444ÂN°pÇãÁ _step‚563 & lr-AdamW/pg1‚7.827277429893752e-05 & lr-AdamW/pg2‚7.827277429893752e-05 trainer/global_step‚6899 % _timestamp‚1.7884219444916391e+09 _runtime‚5444.229226723~³¶ÌäÆ & lr-AdamW/pg2‚7.827277429893752e-05 trainer/global_step‚6899 % _timestamp‚1.7884219444916391e+09 _runtime‚5444.229226723 _step‚563 & lr-AdamW/pg1‚7.827277429893752e-05³AC¡2å- _wandbruntime‚5444  _runtime‚5444ø×g`²æßO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884219444924245e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443395864932864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5750301684818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5771546500608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚6899 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32921259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 E?DeviceStatsMonitor.on_ÎâÍ3train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130099405 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5444.230235928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314…®Ž]ê'çäO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400454339609088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389971718491136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5770890138112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚406225211744256 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚395722037215314 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884219444924245e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054457541632 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389971735530496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5749647247062 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100121888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚406225886109696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395721365738198 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58413662 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91335382 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚443395864932864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437341601246720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230241331 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400454321606144 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100121888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100121886 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32921720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5750301684818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5771546500608 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400454339609088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚6899 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230221291 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400454321606144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32921259 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130119443 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130119443 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770890138112 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100121886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130099405 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚437341577306624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _runtime‚5444.230235928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚443396058788352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58413660 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚406225211744256 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788421945232895e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130117859 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130137923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _runtime‚5444.971121198 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406283058589696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130137923 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395777757781046 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚565 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472<ø7û "êíM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100135902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚406282384210944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91347606 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58421828 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395777757781046 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚565 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚443458564135424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5771709565952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5772365941760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6055312151040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5750463682026 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788421945232895e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130117859 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6055150637568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130137923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390026622621696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5772365941760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100135904 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 _runtime‚5444.971121198 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400510692647936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5771709565952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚406283058589696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130117859 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230273827 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚443458378681856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437403251984384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230253761 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58421830 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚437403228044288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5751118119990 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100135904 XJDeviceSt7Ó WSatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130137923 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32926235 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚457 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844µ·Àt)2ë- _wandbruntime‚5444  _runtime‚5444Šwå@XìÒ 0loss/FBt2/assignment_loss‚0.4208677071845159 .loss/FRt1/detection_loss‚2.255598892970961 /loss/FRt4/detection_loss‚1.5881723480180223 1loss/SRqqt3/detection_loss‚1.8948291174827083 0loss/FRt2/assignment_loss‚3.5410849241649403 2loss/SRqqt3/assignment_loss‚1.3240972327128533 0loss/FBt1/assignment_loss‚0.5694534820504487 % _timestamp‚1.7884219452346368e+09 1loss/FBt4/assignment_loss‚0.12428569934866573 0loss/SRqqt2/detection_loss‚2.378376940318516 .loss/FBt3/detection_loss‚4.365673790375392 $loss/total_loss‚44.6782714774093 0loss/FBt3/assignment_loss‚0.3760170495637188 /loss/FBt1/detection_loss‚2.0882344354282725 2loss/SRqqt1/assignment_loss‚1.4339920080413944 1loss/SRqqt2/assignment_loss‚2.211093487909862 1loss/SRqqt1/detection_loss‚2.4296100500382876 /loss/FBt4/detection_loss‚0.6560996778380985 epoch‚5 0loss/FRt4/assignment_loss‚1.7468304659457916 trainer/global_step‚6899 /loss/FRt1/assignment_loss‚2.716533427335778 0loss/FRt3/assignment_loss‚2.4389068292719975 .loss/FRt2/detection_loss‚2.951496892115649 1loss/SRqqt4/detection_loss‚1.0163296371125259 .loss/FBt2/detection_loss‚2.798789381980896 2loss/SRqqt4/assignment_loss‚1.0066561582068412 /loss/FRt3/detection_loss‚2.3452418419931615 _runtime‚5444.972104244 _step‚566¢®I]í× 0loss/SRqqt2/detection_loss‚2.378376940318516 .loss/FBt3/detection_loss‚4.365673790375392 $loss/total_loss‚44.6782714774093 0loss/FBt3/assignment_loss‚0.3760170495637188 /loss/FBt1/detection_loss‚2.0882344354282725 2loss/SRqqt1/assignment_loss‚1.4339920080413944 1loss/SRqqt2/assignment_loss‚2.211093487909862 1loss/SRqqt1/detection_loss‚2.4296100500382876 /loss/FBt4/detection_loss‚0.6560996778380985 epoch‚5 0loss/FRt4/assignment_loss‚1.7468304659457916 trainer/global_step‚6899 /loss/FRt1/assignment_loss‚2.716533427335778 0loss/FRt3/assignment_loss‚2.4389068292719975 .loss/FRt2/detection_loss‚2.951496892115649 1loss/SRqqt4/detection_loss‚1.0163296371125259 .loss/FBt2/detection_loss‚2.798789381980896 2loss/SRqqt4/assignment_loss‚1.0066561582068412 /loss/FRt3/detection_loss‚2.3452418419931615 _runtime‚5444.972104244 _step‚566 0loss/FBt2/assignment_loss‚0.4208677071845159 .loss/FRt1/detection_loss‚2.255598892970961 /loss/FRt4/detection_loss‚1.5881723480180223 1loss/SRqqt3/detection_loss‚1.8948291174827083 0loss/FRt2/assignment_loss‚3.5410849241649403 2loss/SRqqt3/assignment_loss‚1.3240972327128533 0loss/FBt1/assignment_loss‚0.5694534820504487 % _timestamp‚1.7884219452346368e+09 1loss/FBt4/assignment_loss‚0.12428569934866573¶b@IÒ’î‚(:‡ ¹ÎäÔ¬Ðâ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§§Ëyï‚(:î ¹ÎäÔÞÀ¸€ disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.drbd2265.out‚0# disk.dm-0.out‚23257.78369140625 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 338.30078125 disk.sda2.out‚0 disk.sda3.out‚ 338.30078125! proc.memory.rssMB‚ 2772.390625 cpu‚25.324570003724162* disk./.usagePercent‚17.368932539516095 disk.dm-0.in‚4.671875 disk.drbd1716.out‚0& disk.nvme0n1.out‚23257.78369140625" memory_percent‚90.663743019104) proc.memory.availableMB‚ 1529.65234375$ disk./.usageGB‚258.8096466064453 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 network.sent‚ 1.597244e+07 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚23257.78369140625 disk.sda2.in‚0 network.recv‚ 2.023343e+06 disk.rbd0.in‚0* proc.memory.percent‚16.921329498291016 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda.out‚ 338.30078125 disk.drbd1716.in‚0 disk.rbd1.out‚ 2.759765625 disk.rbd0.out‚0:cñ ð‚(:… ÈÎäÔ›ìˆ~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.01' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$jÅyñ‚(:î ÈÎäԧ䠀 network.recv‚ 2.029854e+06 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 338.6171875 disk.drbd2266.out‚0$ memory_percent‚90.66555500030518* proc.memory.percent‚16.921329498291016 cpu‚25.323821107722633* disk./.usagePercent‚17.369348039744168 disk.nvme0c0n1.in‚4.671875 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda1.out‚0& disk.nvme0n1.out‚23315.86767578125 disk.rbd0.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚23315.86767578125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 338.6171875 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375# disk.dm-0.out‚23315.86767578125 network.sent‚ 1.6081449e+07$ disk./.usageGB‚258.8158378601074 disk.nvme0n1.in‚4.671875) proc.memory.availableMB‚ 1529.35546875! proc.memory.rssMB‚ 2772.390625 disk.sda.out‚ 338.6171875 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375ÈÏ#—’ò‚(:‡ ×ÎäÔàå~ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.084' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.2168 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>è’{tó‚(:é ×ÎäÔ¸®€% disk./.usageGB‚258.82644271850586 disk.dm-0.in‚4.671875# disk.dm-0.out‚23358.77392578125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 cpu‚25.32302926987237 network.sent‚ 1.6086769e+07 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 339.140625 network.recv‚ 2.033537e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625& disk.nvme0n1.out‚23358.77392578125 disk.rbd0.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚23358.77392578125 disk.sda3.out‚ 339.140625 disk.dm-1.out‚ 339.140625$ memory_percent‚90.66400527954102 disk.sda2.out‚0' proc.memory.availableMB‚ 1529.609375* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.370059740689353ÞÁØî2ô- _wandbruntime‚5483  _runtime‚5483h!5ŠÅõ¿ % lr-AdamW/pg1‚7.70590151586215e-05 % lr-AdamW/pg2‚7.70590151586215e-05 trainer/global_step‚6949 % _timestamp‚1.7884219838962545e+09 _runtime‚5483.633782061 _step‚567ûù—ÊöÄ trainer/global_step‚6949 % _timestamp‚1.7884219838962545e+09 _runtime‚5483.633782061 _step‚567 % lr-AdamW/pg1‚7.70590151586215e-05 % lr-AdamW/pg2‚7.70590151586215e-05·ª¶A¤2÷- _wandbruntime‚5483  _runtime‚5483zQðÛä'øÞO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392716925019136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100822686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33149945 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812532306432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100822688 _runtime‚5483.634775185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440424138129920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788421983897005e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91972007 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403271973548544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446521628960768 trainer/global_step‚6949 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5811875943936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131045495 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606¬~9©é'ùãO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚409083849492480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚398508079082722 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _step‚568 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5791137024226 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409084523857920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5811875943936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚446521814427648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231848143 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100822686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58822062 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409083849492480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403271973548544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446521628960768 trainer/global_step‚6949 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5811875943936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131025457 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131045495 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6097676297728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398507407605606 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409084523857920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392716925019136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231868183 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100822686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33149945 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33149495 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6097514770944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5812532306432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5790482586470 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100822688 _runtime‚5483.634775185 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚440424138129920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚440424114189824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788421983897005e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91972007 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403271973548544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5812532306432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131045495 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231880613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/r¬õÆгequested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884219846376958e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231880613 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33154042 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58830228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5484.375640875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131043911 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5812695371776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 % _timestamp‚1.7884219846376958e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91984727 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚440485788867584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131043911 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5813351747584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392771846189056 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚403328326587392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚409141696337920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446584322072576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6098363302912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398563128171130 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5791953459398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231880613 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131063975 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131063975 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚398563799648454 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6098533204992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33154042 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231900679 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58830228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚392771829149696 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446584128230400 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91984270 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403328326587392 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚5484.375640875 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131043911 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100836704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100836704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100836702¹ã°G¿2ý- _wandbruntime‚5484  _runtime‚5484‰6¹8YþÓ 1loss/SRqqt1/detection_loss‚1.9203131363580102 _runtime‚5484.377120658 /loss/FBt4/detection_loss‚0.5300960012402091 trainer/global_step‚6949 /loss/FRt1/assignment_loss‚3.801874795021155 /loss/FRt4/assignment_loss‚1.123876612294804 2loss/SRqqt3/assignment_loss‚1.7800625529378018 2loss/FBt3/assignment_loss‚0.034348973850183026 1loss/SRqqt4/detection_loss‚0.9520858765641849 _step‚570 1loss/SRqqt3/detection_loss‚1.7818655937593033 1loss/FBt4/assignment_loss‚0.24346945354231686 /loss/FRt3/detection_loss‚2.1172501370310783 /loss/FBt1/detection_loss‚1.8328106453021367 -loss/FBt3/detection_loss‚3.06904194355011 % _timestamp‚1.7884219846396296e+09 .loss/FRt3/assignment_loss‚2.88265906388943 /loss/SRqqt2/detection_loss‚2.17152660909821 %loss/total_loss‚43.64592886816168 2loss/SRqqt1/assignment_loss‚1.4826408872674954 0loss/FBt1/assignment_loss‚0.7942605068674311 -loss/FBt2/detection_loss‚2.70707539220651 2loss/SRqqt2/assignment_loss‚1.9999747672940003 /loss/FRt2/detection_loss‚3.1959777291004476 /loss/FRt4/detection_loss‚2.0026029370386493 epoch‚5 0loss/FRt2/assignment_loss‚3.4755621598317075 2loss/SRqqt4/assignment_loss‚1.2022974361975989 1loss/FBt2/assignment_loss‚0.22382994118864494 /loss/FRt1/detection_loss‚2.3204257167302647ìX’Ç^ÿØ 0loss/FRt2/assignment_loss‚3.4755621598317075 2loss/SRqqt4/assignment_loss‚1.2022974361975989 1loss/FBt2/assignment_loss‚0.22382994118864494 /loss/FRt1/detection_loss‚2.3204257167302647 1loss/SRqqt1/detection_loss‚1.9203131363580102 _runtime‚5484.377120658 /loss/FBt4/detection_loss‚0.5300960012402091 trainer/global_step‚6949 /loss/FRt1/assignment_loss‚3.801874795021155 /loss/FRt4/assignment_loss‚1.123876612294804 2loss/SRqqt3/assignment_loss‚1.7800625529378018 2loss/FBt3/assignment_loss‚0.034348973850183026 1loss/SRqqt4/detection_loss‚0.9520858765641849 _step‚570 1loss/SRqqt3/detection_loss‚1.7818655937593033 1loss/FBt4/assignment_loss‚0.24346945354231686 /loss/FRt3/detection_loss‚2.1172501370310783 /loss/FBt1/detection_loss‚1.8328106453021367 -loss/FBt3/detection_loss‚3.06904194355011 % _timestamp‚1.7884219846396296e+09 .loss/FRt3/assignment_loss‚2.88265906388943 /loss/SRqqt2/detection_loss‚2.17152660909821 %loss/total_loss‚43.64592886816168 2loss/SRqqt1/assignment_loss‚1.4826408872674954 0loss/FBt1/assignment_loss‚0.7942605068674311 -loss/FBt2/detection_loss‚2.70707539220651 2loss/SRqqt2/assignment_loss‚1.9999747672940003 /loss/FRt2/detection_loss‚3.1959777291004476 /loss/FRt4/detection_loss‚2.0026029370386493 epoch‚5º7Y…€‚(:’ æÎäÔ»ò¤~ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.721' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.944200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h/iÜr‚(:ç æÎäÔíÁ disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.out‚ 339.5390625 disk.drbd2265.out‚0$ memory_percent‚90.66634178161621 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.6092893e+07 network.recv‚ 2.03814e+06$ disk./.usageGB‚258.8191452026367 disk.rbd0.out‚0% disk.nvme0n1.out‚23410.5634765625" disk.dm-0.out‚23410.5634765625 disk.sda.in‚ 126.77734375) disk./.usagePercent‚17.36956999827635 disk.dm-1.out‚ 339.5390625 disk.drbd1716.out‚0( proc.memory.availableMB‚ 1529.2265625! proc.memory.rssMB‚ 2772.390625 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 2.759765625* proc.memory.percent‚16.921329498291016 cpu‚25.32227849982756 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚23410.5634765625 disk.sda2.in‚0 disk.sda.out‚ 339.5390625 disk.drbd1716.in‚0-if‘‚‚(:† õÎäÔôïº~ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.991 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™© £oƒ‚(:ä õÎäÔ踀 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0# disk.nvme0n1.out‚23441.44140625 disk.rbd0.out‚0 disk.dm-0.in‚4.671875! proc.memory.rssMB‚ 2772.390625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 340.1015625 disk.nvme0c0n1.in‚4.671875 disk.dm-1.out‚ 340.1015625 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-0.out‚23441.44140625( proc.memory.availableMB‚ 1529.2265625* proc.memory.percent‚16.921329498291016 cpu‚25.321546359877416 disk.drbd2265.out‚0 network.sent‚ 1.6201674e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚23441.44140625 disk.sda3.out‚ 340.1015625 network.recv‚ 2.044357e+06$ disk./.usageGB‚258.8365707397461 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚90.66634178161621* disk./.usagePercent‚17.370739440692763 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0‹«ÁÞ’„‚(:‡ „ÏäÔ¼” ~ gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÞO"y…‚(:î „ÏäÔ—“À€ disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2772.390625 disk.rbd1.out‚ 2.759765625 disk.sda2.out‚0 disk.sda3.out‚ 340.55078125 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875# disk.dm-0.out‚23510.73779296875$ memory_percent‚90.66486358642578 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23510.73779296875 network.sent‚ 1.6208121e+07$ disk./.usageGB‚258.8325424194336 disk.drbd1716.in‚0 disk.rbd0.in‚0 network.recv‚ 2.049304e+06* disk./.usagePercent‚17.370469096736603 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 340.55078125& disk.nvme0n1.out‚23510.73779296875 disk.drbd2080.out‚0 cpu‚25.320811367332404 disk.sda.out‚ 340.55078125 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0& proc.memory.availableMB‚ 1529.46875* proc.memory.percent‚16.9213294982910162‰&T2†- _wandbruntime‚5522  _runtime‚5522øFCELJÁ _step‚571 & lr-AdamW/pg1‚7.584677405305507e-05 & lr-AdamW/pg2‚7.584677405305507e-05 trainer/global_step‚6999 % _timestamp‚1.7884220229821424e+09 _runtime‚5522.719698491å¬ÂÌˆÆ trainer/global_step‚6999 % _timestamp‚1.7884220229821424e+09 _runtime‚5522.719698491 _step‚571 & lr-AdamW/pg1‚7.584677405305507e-05 & lr-AdamW/pg2‚7.584677405305507e-05»×%«ï2‰- _wandbruntime‚5522  _runtime‚5522’þWYïŠßO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5852861749760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443506675013120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92608201 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚572 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5522.720666638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233474995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59230462 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59230460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496û/öê'‹äO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59230462 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59230460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33377282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395462148586496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5852861749760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449647679341056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚443506675013120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚411942487240704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233495035 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92608201 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 _step‚572 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚406089625490944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚5522.720666638 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449647493874176 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚411942487240704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚411943161606144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101523486 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131951509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5853518112256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚443506651073024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233474995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5852861749760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6141004327936 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101523488 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131971547 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚406089643493888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚401293449473014 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92607742 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395462131547136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6140842801152 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚6999 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884220237254856e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33382344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443568301810688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5853681177600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406145996532736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406145996532736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832788798806 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0„ Yfô&ŽîM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449710196056576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5854337553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59238630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412000334086144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5853681177600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233527531 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚401349841515862 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131990027 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92620511 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449710014797312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884220237254856e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233527531 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33382344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59238628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚443568301810688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5853681177600 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395517052717056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101537504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101537502 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406145996532736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395517035677696 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406145996532736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6141712986624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233507465 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832788798806 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406145978529792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92620974 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33381883 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131969963 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚6999 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101537502½ïÀùh2- _wandbruntime‚5523  _runtime‚5523÷#­¨Ô _runtime‚5523.465153606 1loss/SRqqt1/detection_loss‚2.1885604222078583 1loss/SRqqt3/detection_loss‚1.7609950185083507 /loss/FRt2/assignment_loss‚½¢G²3.242045449799505 1loss/SRqqt2/assignment_loss‚1.991315750895362 $loss/total_loss‚43.5429567207765 % _timestamp‚1.7884220237276568e+09 2loss/SRqqt1/assignment_loss‚1.6933345597338032 2loss/SRqqt4/assignment_loss‚1.1308768172366697 1loss/FBt1/assignment_loss‚0.30943608029999514 1loss/FBt4/assignment_loss‚0.16550618977830076 /loss/FBt3/detection_loss‚2.3847053315904407 .loss/FRt2/detection_loss‚2.933819994844239 /loss/FRt3/assignment_loss‚2.600288836793466 2loss/SRqqt3/assignment_loss‚1.2371164320461583 _step‚574 /loss/FRt1/assignment_loss‚3.481420330311123 /loss/FRt4/assignment_loss‚1.652825024511133 /loss/FRt4/detection_loss‚1.5937913374177046 /loss/FBt4/detection_loss‚0.5707580004474783 1loss/FBt2/assignment_loss‚0.29289431292561974 0loss/FBt3/assignment_loss‚0.5162374200428732 .loss/FRt3/detection_loss‚2.438005781986497 1loss/SRqqt4/detection_loss‚1.3300165399054036 /loss/FRt1/detection_loss‚2.3852036258107736 1loss/SRqqt2/detection_loss‚2.3332875684687964 /loss/FBt1/detection_loss‚2.3027943101796238 /loss/FBt2/detection_loss‚3.0077215850353243 epoch‚5 trainer/global_step‚6999Àö¦_‘Ù 1loss/SRqqt2/detection_loss‚2.3332875684687964 /loss/FBt1/detection_loss‚2.3027943101796238 /loss/FBt2/detection_loss‚3.0077215850353243 epoch‚5 trainer/global_step‚6999 _runtime‚5523.465153606 1loss/SRqqt1/detection_loss‚2.1885604222078583 1loss/SRqqt3/detection_loss‚1.7609950185083507 /loss/FRt2/assignment_loss‚3.242045449799505 1loss/SRqqt2/assignment_loss‚1.991315750895362 $loss/total_loss‚43.5429567207765 % _timestamp‚1.7884220237276568e+09 2loss/SRqqt1/assignment_loss‚1.6933345597338032 2loss/SRqqt4/assignment_loss‚1.1308768172366697 1loss/FBt1/assignment_loss‚0.30943608029999514 1loss/FBt4/assignment_loss‚0.16550618977830076 /loss/FBt3/detection_loss‚2.3847053315904407 .loss/FRt2/detection_loss‚2.933819994844239 /loss/FRt3/assignment_loss‚2.600288836793466 2loss/SRqqt3/assignment_loss‚1.2371164320461583 _step‚574 /loss/FRt1/assignment_loss‚3.481420330311123 /loss/FRt4/assignment_loss‚1.652825024511133 /loss/FRt4/detection_loss‚1.5937913374177046 /loss/FBt4/detection_loss‚0.5707580004474783 1loss/FBt2/assignment_loss‚0.29289431292561974 0loss/FBt3/assignment_loss‚0.5162374200428732 .loss/FRt3/detection_loss‚2.438005781986497 1loss/SRqqt4/detection_loss‚1.3300165399054036 /loss/FRt1/detection_loss‚2.3852036258107736¾S~ñ$’‚(:’ “ÏäÔï¾®~ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚151.791' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.358200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥<ðz“‚(:ï “ÏäÔû­à€ network.recv‚ 2.05265e+06 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 341.21484375 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 cpu‚25.320078528450892$ disk./.usageGB‚258.8452796936035 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda.out‚ 341.21484375$ memory_percent‚90.66522121429443 disk.rbd0.out‚0( disk.nvme0c0n1.out‚23553.79638671875 disk.drbd1716.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.371323905893433 disk.sda3.out‚ 341.21484375* proc.memory.percent‚16.921329498291016 network.sent‚ 1.621339e+07 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1529.41015625 disk.drbd2265.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚23553.79638671875 disk.rbd1.in‚0& disk.nvme0n1.out‚23553.79638671875•Ç¥u”‚(:ê ¢ÏäÔ¤º§€* disk./.usagePercent‚17.371734541978352 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚23603.7626953125$ disk./.usageGB‚258.8513984680176" disk.dm-0.out‚23603.7626953125 disk.nvme0c0n1.in‚4.671875$ memory_percent‚90.66524505615234! proc.memory.rssMB‚ 2772.390625 disk.dm-0.in‚4.671875 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.058733e+06 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.759765625 disk.sda2.out‚0 disk.sda3.out‚ 341.72265625 cpu‚25.31931140314598 network.sent‚ 1.6322092e+07 disk.dm-1.out‚ 341.72265625 disk.sda.out‚ 341.72265625% disk.nvme0n1.out‚23603.7626953125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375& proc.memory.availableMB‚ 1529.40625* proc.memory.percent‚16.921329498291016 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0B“•‚(:ˆ ¢ÏäÔ—‹š gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚129.489' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ˆ¬Œ2–- _wandbruntime‚5561  _runtime‚5561:ÌHSÅ—¿ % lr-AdamW/pg1‚7.46363036114676e-05 % lr-AdamW/pg2‚7.46363036114676e-05 trainer/global_step‚7049 % _timestamp‚1.7884220621720889e+09 _runtime‚5561.909609832 _step‚575}ú™Ê˜Ä _runtime‚5561.909609832 _step‚575 % lr-AdamW/pg1‚7.46363036114676e-05 % lr-AdamW/pg2‚7.46363036114676e-05 trainer/global_step‚7049 % _timestamp‚1.7884220621720889e+09¿e$!72™- _wandbruntime‚5561  _runtime‚5561úÖÝå'šßO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446589187956224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398207355114496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5893847555584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93243461 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5872807703042 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414801124988928 _runtime‚5561.910556088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5894503918080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446589211896320 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102224286 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132877561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132897599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚404080162817538 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452773227724288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404079491340422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408907295436288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884220621727927e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 trainer/global_step‚7049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33604599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132897599 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452773227724288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132877561 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚404079491340422 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408907295436288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 % _timestamp‚1.7884220621727927e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408907277433344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 trainer/global_step‚7049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6184039768064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33604599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414801124988928 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184182420480 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408907295436288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚414801799354368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5872153265286 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132877561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908À‰¡*Þ2œ- _wandbruntime‚5562  _runtime‚5562‹ YåzçM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408963648475136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 $ _timestamp‚1.788422062905567e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132896015 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398262259245056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102238302 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102238302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132916079 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404135211905946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitorCQ‰»s.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404135883383270 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚ 5562.64360911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835726230016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872^e&ò&žìM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _runtime‚ 5562.64360911 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452835726230016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚408963630472192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235154383 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414858971834368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446650862633984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _step‚577 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59647028 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408963648475136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚447 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33608596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6185055341056 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414858971834368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33609041 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6184887536128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235134317 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452835917975040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408963648475136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102238304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5894666983424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414858297455616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 $ _timestamp‚1.788422062905567e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132896015 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398262259245056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235134317 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚408963630472192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59647030 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102238302 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102238302 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132916079 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚404135211905946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235154383 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5873624138214 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5895323359232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5872969700250 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93256071 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93255624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446650838693888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404135883383270ÁýÖ³”2Ÿ- _wandbruntime‚5562  _runtime‚5562PŸŠ\ Ö /loss/FRt2/detection_loss‚3.1112176116023744 /loss/FRt4/detection_loss‚1.6371657035499811 % _timestamp‚1.7884220629075117e+09 0loss/FRt1/assignment_loss‚3.4609937929525607 0loss/FRt2/assignment_loss‚3.6590149104595184 1loss/FBt3/assignment_loss‚0.10380007504967839 1loss/SRqqt3/detection_loss‚1.5349215737591355 -loss/FBt2/detection_loss‚2.85411331483296 /loss/FBt4/detection_loss‚0.5162791072629219 epoch‚5 _step‚578 2loss/SRqqt1/assignment_loss‚1.4029050166177195 /loss/FRt1/detection_loss‚2.3886719869404303 .loss/FRt3/detection_loss‚2.217742537312648 %loss/total_loss‚42.79420931973095 _runtime‚5562.644973265 1loss/SRqqt2/assignment_loss‚1.963228998084863 1loss/FBt4/assignment_loss‚0.16773416693331622 1loss/SRqqt1/detection_loss‚1.9251425356365914 0loss/FRt3/assignment_loss‚2.9251929802053116 .loss/FRt4/assignment_loss‚1.18340008020401 1loss/SRqqt4/detection_loss‚1.1073904883910397 /loss/FBt1/detection_loss‚2.5370570204474707 0loss/FBt1/assignment_loss‚0.4096202018924735 /loss/FBt3/detection_loss‚2.6428846205983842 trainer/global_step‚7049 1loss/SRqqt2/detection_loss‚2.2660575111707053 2loss/SRqqt3/assignment_loss‚1.7641826137205807 2loss/SRqqt4/assignment_loss‚0.6972918509520957 1loss/FBt2/assignment_loss‚0.31820062115418424þ£ aa¡Û 1loss/SRqqt1/detection_loss‚1.9251425356365914 0loss/FRt3/assignment_loss‚2.9251929802053116 .loss/FRt4/assignment_loss‚1.18340008020401 1loss/SRqqt4/detection_loss‚1.1073904883910397 /loss/FBt1/detection_loss‚2.5370570204474707 0loss/FBt1/assignment_loss‚0.4096202018924735 /loss/FBt3/detection_loss‚2.6428846205983842 trainer/global_step‚7049 1loss/SRqqt2/detection_loss‚2.2660575111707053 2loss/SRqqt3/assignment_loss‚1.7641826137205807 2loss/SRqqt4/assignment_loss‚0.6972918509520957 1loss/FBt2/assignment_loss‚0.31820062115418424 /loss/FRt2/detection_loss‚3.1112176116023744 /loss/FRt4/detection_loss‚1.6371657035499811 % _timestamp‚1.7884220629075117e+09 0loss/FRt1/assignment_loss‚3.4609937929525607 0loss/FRt2/assignment_loss‚3.6590149104595184 1loss/FBt3/assignment_loss‚0.10380007504967839 1loss/SRqqt3/detection_loss‚1.5349215737591355 -loss/FBt2/detection_loss‚2.85411331483296 /loss/FBt4/detection_loss‚0.5162791072629219 epoch‚5 _step‚578 2loss/SRqqt1/assignment_loss‚1.4029050166177195 /loss/FRt1/detection_loss‚2.3886719869404303 .loss/FRt3/detection_loss‚2.217742537312648 %loss/total_loss‚42.79420931973095 _runtime‚5562.644973265 1loss/SRqqt2/assignment_loss‚1.963228998084863 1loss/FBt4/assignment_loss‚0.16773416693331622Â’x¢‚(:’ ±ÏäÔ†Åý} gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚131.141' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.228199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08¿ÐÉu£‚(:ê ±ÏäÔÅ„Ÿ€ disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 network.sent‚ 1.6328e+07 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚23645.13818359375 disk.nvme0c0n1.in‚4.671875 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 342.5703125 disk.sda1.out‚0) proc.memory.availableMB‚ 1529.37890625( disk.nvme0c0n1.out‚23645.13818359375) disk./.usagePercent‚17.37228009403382 disk.rbd0.out‚0# disk.dm-0.out‚23645.13818359375 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.921329498291016 cpu‚25.318589522878007 disk.drbd2265.out‚0 disk.sda2.out‚0$ disk./.usageGB‚258.8595275878906 disk.drbd1716.out‚0 disk.sda3.out‚ 342.5703125 disk.sda1.in‚0$ memory_percent‚90.66541194915771 network.recv‚ 2.062822e+06 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.759765625 disk.sda.out‚ 342.5703125 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625h@Ë夂(:’ ÀÏäÔㆿ~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.062400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â7Í®{¥‚(:ð ÀÏäÔ©§Ô€ disk.drbd2080.in‚0% disk./.usageGB‚258.87760162353516 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 342.86328125$ memory_percent‚90.66693782806396 cpu‚25.31786692686042 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0* proc.memory.percent‚16.921329498291016 disk.nvme0n1.in‚4.671875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.759765625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.out‚ 342.86328125# disk.dm-0.out‚23699.90869140625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 342.86328125 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚4.671875& disk.nvme0n1.out‚23699.90869140625( disk.nvme0c0n1.out‚23699.90869140625 disk.drbd2080.out‚0 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1529.12890625! proc.memory.rssMB‚ 2772.390625 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.6437039e+07 network.recv‚ 2.069578e+06) disk./.usagePercent‚17.37349305773105u“â’¦‚(:‡ ÏÏäÔ˜ô­~ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OyY%}§‚(:ò ÏÏäÔ…åÜ& disk.nvme0n1.out‚23742.75439453125 disk.rbd0.out‚0 disk.sda3.out‚ 343.37890625 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 network.sent‚ 1.6442325e+07 disk.rbd1.in‚0 disk.drbd1716.in‚0$ memory_percent‚90.66693782806396! proc.memory.rssMB‚ 2772.390625* proc.memory.percent‚16.921329498291016 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 343.37890625 disk.nvme0n1.in‚4.671875 cpu‚25.317142218754018 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚23742.75439453125 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚23742.75439453125 proc.cpu.threads‚17% disk./.usageGB‚258.88094329833984 disk.rbd1.out‚ 2.759765625 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚ 2.072976e+06 disk.dm-1.out‚ 343.37890625) proc.memory.availableMB‚ 1529.12890625* disk./.usagePercent‚17.373717320331046 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0éØ‘Q2¨- _wandbruntime‚5601  _runtime‚5601PgAÙÇ©Á _runtime‚5601.106013499 _step‚579 & lr-AdamW/pg1‚7.342785609408489e-05 & lr-AdamW/pg2‚7.342785609408489e-05 trainer/global_step‚7099 % _timestamp‚1.7884221013683915e+09œ<îeÌªÆ % _timestamp‚1.7884221013683915e+09 _runtime‚5601.106013499 _step‚579 & lr-AdamW/pg1‚7.342785609408489e-05 & lr-AdamW/pg2‚7.342785609408489e-05 trainer/global_step‚7099ï2«- _wandbruntime‚5601  _runtime‚5601VÄùã'¬ÝO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236728699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400952561642496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚580 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411724929375744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417660437102592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102925088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33832777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5934833361408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚7099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚5601.106886146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411724929375744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93880039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60047262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚441 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚406866204684946 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚406865533207830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5912988604694 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102925086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5913643042450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33832777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚455898825288704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5934833361408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚7099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411724947378688 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5935489723904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚5601.106886146 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417660437102592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133803613 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚439 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236728699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚400952561642496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417659762737152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236748739 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚455899012852736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚417659762737152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6227264073216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _step‚580 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133823651 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102925086 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5934833361408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414Äq*ò2®- _wandbruntime‚5602  _runtime‚5602Ú¹óGï&¯éM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6228126648320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5602.072191244 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133842131 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 trainer/global_step‚7099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411781282414592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6227967232000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102939102 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884221023340828e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944%ùAô&°îM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236781235 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5935652789248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5935652789248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236761169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚449733375577088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60055428 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6227967232000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5936309165056 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102939102 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5936309165056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93892367 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33837387 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33836939 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102939102 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚417717609582592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚401007448733696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417716935203840 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884221023340828e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93892817 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚406921925250678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133822067 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411781300417536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102939104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6228126648320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚411781282414592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236781235 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417717609582592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚411781300417536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚5602.072191244 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133842131 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚449733399517184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚401007465773056 trainer/global_step‚7099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236761169 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚411781282414592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ÅéZ1Ñ2±- _wandbruntime‚5602  _runtime‚5602ä#_VT²Î 0loss/FBt1/assignment_loss‚0.3083984775073784 _runtime‚5602.073341846 # _timestamp‚1.78842210233585e+09 1loss/SRqqt2/assignment_loss‚2.277735867856869 /loss/FRt1/detection_loss‚2.2646282321097804 1loss/SRqqt4/detection_loss‚0.8782387923287309 /loss/FBt1/detection_loss‚2.8876267539130316 -loss/FBt3/detection_loss‚2.26198433978217 trainer/global_step‚7099 0loss/FBt2/assignment_loss‚0.6954615660272988 epoch‚5 /loss/FRt4/assignment_loss‚1.698003113269806 0loss/FBt4/assignment_loss‚0.1500539146069705 1loss/SRqqt3/detection_loss‚1.7442443698644639 _step‚582 2loss/SRqqt3/assignment_loss‚1.7099526401947844 1loss/FBt3/assignment_loss‚0.30823469813913107 /loss/FBt2/detection_loss‚2.5917260348796844 2loss/SRqqt1/assignment_loss‚1.2237908021595918 1loss/SRqqt1/detection_loss‚2.1214587300231584 1loss/SRqqt2/detection_loss‚2.1723609004701885 /loss/FBt4/detection_loss‚0.5398684736324607 %loss/total_loss‚44.47236193347808 /loss/FRt1/assignment_loss‚2.933767194841422 0loss/FRt3/assignment_loss‚3.5306635778777453 -loss/FRt2/detection_loss‚3.42179580710151 .loss/FRt3/detection_loss‚2.177068771756425 /loss/FRt4/detection_loss‚1.8212367735936172 /loss/FRt2/assignment_loss‚3.868095099925995 1loss/SRqqt4/assignment_loss‚0.885967001615875Á»ðŠY³Ó /loss/FRt1/assignment_loss‚2.933767194841422 0loss/FRt3/assignment_loss‚3.5306635778777453 -loss/FRt2/detection_loss‚3.42179580710151 .loss/FRt3/detection_loss‚2.177068771756425 /loss/FRt4/detection_loss‚1.8212367735936172 /loss/FRt2/assignment_loss‚3.868095099925995 1loss/SRqqt4/assignment_loss‚0.885967001615875 0loss/FBt1/assignment_loss‚0.3083984775073784 _runtime‚5602.073341846 # _timestamp‚1.78842210233585e+09 1loss/SRqqt2/assignment_loss‚2.277735867856869 /loss/FRt1/detection_loss‚2.2646282321097804 1loss/SRqqt4/detection_loss‚0.8782387923287309 /loss/FBt1/detection_loss‚2.8876267539130316 -loss/FBt3/detection_loss‚2.26198433978217 trainer/global_step‚7099 0loss/FBt2/assignment_loss‚0.6954615660272988 epoch‚5 /loss/FRt4/assignment_loss‚1.698003113269806 0loss/FBt4/assignment_loss‚0.1500539146069705 1loss/SRqqt3/detection_loss‚1.7442443698644639 _step‚582 2loss/SRqqt3/assignment_loss‚1.7099526401947844 1loss/FBt3/assignment_loss‚0.30823469813913107 /loss/FBt2/detection_loss‚2.5917260348796844 2loss/SRqqt1/assignment_loss‚1.2237908021595918 1loss/SRqqt1/detection_loss‚2.1214587300231584 1loss/SRqqt2/detection_loss‚2.1723609004701885 /loss/FBt4/detection_loss‚0.5398684736324607 %loss/total_loss‚44.47236193347808ƯÞ,´‚(:‡ ÞÏäÔû·ü} gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚èª`Ä24.7882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O¥nnµ‚(:ã ÞÏäÔ‚ˆœ€ disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 cpu‚25.316489654080918 disk.drbd2080.out‚0 disk.sda.out‚ 344.171875$ disk.nvme0n1.out‚23798.353515625 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.6448301e+07 disk.dm-1.out‚ 344.171875 proc.cpu.threads‚17 network.recv‚ 2.0775e+06 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 2.845703125 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.sda3.out‚ 344.171875 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚23798.353515625$ memory_percent‚90.66565036773682* proc.memory.percent‚16.921329498291016$ disk./.usageGB‚258.8914680480957 disk.drbd2080.in‚0! disk.dm-0.out‚23798.353515625 disk.sda1.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1529.33984375* disk./.usagePercent‚17.374423645118014[¶‚(:’ íÏäÔá’µ~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.339' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.867799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾78ßr·‚(:ç íÏäÔ¡ø°€* disk./.usagePercent‚17.375171187117996$ disk./.usageGB‚258.9026069641113 disk.drbd2080.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.921329498291016 disk.dm-0.in‚4.671875 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚23842.0869140625 disk.sda2.in‚0$ memory_percent‚90.66581726074219 disk.rbd0.in‚0' disk.nvme0c0n1.out‚23842.0869140625 disk.dm-1.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17 network.recv‚ 2.083723e+06 disk.sda.in‚ 126.77734375 disk.sda.out‚ 344.7109375 disk.rbd0.out‚0 network.sent‚ 1.6557227e+07 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 1529.3125 disk.drbd2266.in‚0" disk.dm-0.out‚23842.0869140625 disk.dm-1.out‚ 344.7109375 disk.rbd1.out‚ 2.908203125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 344.7109375 cpu‚25.315780730298986£Ë’¸‚(:‡ üÏäÔû¹„~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ Q°w¹‚(:ì üÏäÔ€™€( disk.nvme0c0n1.out‚23896.20361328125 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda3.out‚ 345.3203125 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 345.3203125% disk./.usageGB‚258.91004943847656 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.6563517e+07 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚23896.20361328125# disk.dm-0.out‚23896.20361328125 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.921329498291016 cpu‚25.315078919240268 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0( disk./.usagePercent‚17.3756706578173 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875) proc.memory.availableMB‚ 1529.30859375 proc.cpu.threads‚17 network.recv‚ 2.088432e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 2.908203125 disk.sda.out‚ 345.3203125# memory_percent‚90.6658411026001œ–h2º- _wandbruntime‚5640  _runtime‚5640++±Ç»Á % _timestamp‚1.7884221405468593e+09 _runtime‚5640.284384908 _step‚583 & lr-AdamW/pg1‚7.222168333955828e-05 & lr-AdamW/pg2‚7.222168333955828e-05 trainer/global_step‚7149gŽæXÌ¼Æ & lr-AdamW/pg1‚7.222168333955828e-05 & lr-AdamW/pg2‚7.222168333955828e-05 trainer/global_step‚7149 % _timestamp‚1.7884221405468593e+09 _runtime‚5640.284384908 _step‚583ÇgWpr2½- _wandbruntime‚5640  _runtime‚5640<%¤å'¾ßO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414542581318144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403697768170496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409651575075238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420519074850816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚5640.285561308 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚459024420771328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103625886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414r¿Yê'¿äO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414542599321088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420518400485376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚459024420771328 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5953823944102 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60455660 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚452754261722624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103625886 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚452754285662720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414542581318144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34060716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238375591 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403697768170496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420519074850816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409651575075238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5975819167232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420519074850816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134749703 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134729665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _runtime‚5640.285561308 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5976475529728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5975819167232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6270159048704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Èıc2À- _wandbruntime‚5641  _runtime‚5641_“.]œÁèM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409707295640762 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103639904 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134768183 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5641.011146859 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60463830 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 trainer/global_step‚7149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 $ _timestamp‚1.788422141273115e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134748119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5976638595072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚585 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatäÑjFR sMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Þ#3pó&ÂíM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103639902 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420575572952064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414598934356992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103639904 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚409707295640762 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414598952359936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103639904 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420576247330816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134768183 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 238388021 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94529101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5641.011146859 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60463830 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚452815912460288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238388021 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6271184113152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955294817030 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5954640379066 trainer/global_step‚7149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238408087 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5976638595072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚459086926671360 $ _timestamp‚1.788422141273115e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238408087 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103639902 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 134748119 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5976638595072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _step‚585É°Æ)2Ã- _wandbruntime‚5641  _runtime‚5641¼\V~^ÄØ % _timestamp‚1.7884221412752306e+09 _step‚586 2loss/SRqqt3/assignment_loss‚1.3476692059775814 2loss/SRqqt4/assignment_loss‚1.6043614056892692 1loss/FBt1/assignment_loss‚0.38559744991337086 1loss/FBt2/assignment_loss‚0.40362095683813093 0loss/FBt3/assignment_loss‚0.3386769324541092 /loss/FRt2/detection_loss‚3.0799255106184216 /loss/FRt4/detection_loss‚1.9511928656862843 1loss/SRqqt1/detection_loss‚2.0985134579241276 0loss/FRt2/assignment_loss‚3.6379260001359164 /loss/FRt3/assignment_loss‚3.050969563779377 2loss/FBt4/assignment_loss‚0.057592363837743715 1loss/SRqqt2/detection_loss‚2.3593567467871166 1loss/SRqqt3/detection_loss‚1.5968726070132107 0loss/SRqqt4/detection_loss‚1.176407005265355 /loss/FBt4/detection_loss‚0.5017247249055893 %loss/total_loss‚44.43873333948346 /loss/FRt1/detection_loss‚2.7342213751638638 2loss/SRqqt2/assignment_loss‚1.7187796687441212 .loss/FBt2/detection_loss‚2.742845690250397 .loss/FBt3/detection_loss‚2.603751355409622 /loss/FRt3/detection_loss‚2.0199662302694623 /loss/FRt1/assignment_loss‚3.713153542841182 2loss/SRqqt1/assignment_loss‚1.4680910697206855 /loss/FBt1/detection_loss‚2.2089946925640107 _runtime‚5641.012709048 0loss/FRt4/assignment_loss‚1.6385229176945157 epoch‚5 trainer/global_step‚7149˜áÜÚcÅÝ /loss/FRt1/assignment_loss‚3.713153542841182 2loss/SRqqt1/assignment_loss‚1.4680910697206855 /loss/FBt1/detection_loss‚2.2089946925640107 _runtime‚5641.012709048 0loss/FRt4/assignment_loss‚1.6385229176945157 epoch‚5 trainer/global_step‚7149 % _timestamp‚1.7884221412752306e+09 _step‚586 2loss/SRqqt3/assignment_loss‚1.3476692059775814 2loss/SRqqt4/assignment_loss‚1.6043614056892692 1loss/FBt1/assignment_loss‚0.38559744991337086 1loss/FBt2/assignment_loss‚0.40362095683813093 0loss/FBt3/assignment_loss‚0.3386769324541092 /loss/FRt2/detection_loss‚3.0799255106184216 /loss/FRt4/detection_loss‚1.9511928656862843 1loss/SRqqt1/detection_loss‚2.0985134579241276 0loss/FRt2/assignment_loss‚3.6379260001359164 /loss/FRt3/assignment_loss‚3.050969563779377 2loss/FBt4/assignment_loss‚0.057592363837743715 1loss/SRqqt2/detection_loss‚2.3593567467871166 1loss/SRqqt3/detection_loss‚1.5968726070132107 0loss/SRqqt4/detection_loss‚1.176407005265355 /loss/FBt4/detection_loss‚0.5017247249055893 %loss/total_loss‚44.43873333948346 /loss/FRt1/detection_loss‚2.7342213751638638 2loss/SRqqt2/assignment_loss‚1.7187796687441212 .loss/FBt2/detection_loss‚2.742845690250397 .loss/FBt3/detection_loss‚2.603751355409622 /loss/FRt3/detection_loss‚2.0199662302694623Ê#÷J½Æ‚(:’ ‹ÐäÔÞˆ¡~ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÞyÇ‚(:î ‹ÐäÔĪŽ disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 345.7734375% disk./.usageGB‚258.88841247558594# disk.dm-0.out‚23936.84423828125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.drbd1716.out‚0) disk./.usagePercent‚17.37421858308309 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.9404296875 disk.drbd1716.in‚0* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 network.recv‚ 2.092104e+06 disk.drbd2265.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1529.26953125 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚23936.84423828125& disk.nvme0n1.out‚23936.84423828125 network.sent‚ 1.6568944e+07 disk.dm-1.out‚ 345.7734375 disk.sda.out‚ 345.7734375! proc.memory.rssMB‚ 2772.390625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# memory_percent‚90.6660795211792 cpu‚25.314370637197175ÒLéÛÈ‚(:’ šÐäÔ¦†¶~ gpu.0.gpu‚42.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.338' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.867599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U…¶’sÉ‚(:è šÐäÔÀ§Ã€ disk.sda1.out‚0* disk./.usagePercent‚17.375028078906354 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 346.2890625 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 126.77734375 network.sent‚ 1.6677675e+07 network.recv‚ 2.098749e+06% disk./.usageGB‚258.90047454833984 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚23985.228515625' proc.memory.availableMB‚ 1529.265625! proc.memory.rssMB‚ 2772.390625* proc.memory.percent‚16.921329498291016 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 2.9404296875 disk.drbd1716.in‚0 disk.sda3.out‚ 346.2890625 cpu‚25.313679711151966 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 346.2890625$ disk.nvme0n1.out‚23985.228515625 disk.nvme0c0n1.in‚4.671875$ memory_percent‚90.66610336303711 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚23985.228515625 ”Ä2Ê- _wandbruntime‚5679  _runtime‚5679Ô³ÅË¿ _runtime‚ 5679.20442375 _step‚587 & lr-AdamW/pg1‚7.101803671248193e-05 & lr-AdamW/pg2‚7.101803671248193e-05 trainer/global_step‚7199 $ _timestamp‚1.788422179466851e+09ç6ÙÊÌÄ _runtime‚ 5679.20442375 _step‚587 & lr-AdamW/pg1‚7.101803671248193e-05 & lr-AdamW/pg2‚7.101803671248193e-05 trainer/global_step‚7199 $ _timestamp‚1.788422179466851e+09ËæÊþÞ2Í- _wandbruntime‚5679  _runtime‚5679XN‘å'ÎßO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423377038233600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6016804973056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884221794676507e+09 _step‚588 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60864062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95152997 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104326686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6017461335552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455836822545920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455836798605824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313414849536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚5679.205487017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5994659283510 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135655717 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60864060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6016804973056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430°¼¢Ca ÏäO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104326686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6017461335552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6017461335552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423377712599040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135675755 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462150037248512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚455836822545920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚455836798605824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104326688 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6313414849536 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5995313721266 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚5679.205487017 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5994659283510 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135655717 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60864060 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMoni€7¡‰tor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423377038233600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6016804973056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239982403 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚406442974698496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423377038233600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135675755 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6016804973056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884221794676507e+09 _step‚588 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462150237395456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135655717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚60864062 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34289387 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95152997Ìâ 82Ð- _wandbruntime‚5679  _runtime‚5679Z„^ï&ÑéM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240014873 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455898449343488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018280776704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018280776704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5679.917329932 _step‚589 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6017624400896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104340702 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚417416586299392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚417416604302336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5995475718474 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412493337508170 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5996130156438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135674171 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚423434885079040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240034939 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95165660 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚455898449343488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135694235 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6018280776704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135674171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60872228 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95166106 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6018280776704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚417416604302336 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5679.917329932 _step‚589 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6017624400896 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104340702 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6314274786816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240034939 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34293876 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61272460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5717.998719557 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6058447141376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6356329098240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95789116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6036149060674 $ _timestamp‚1.788422218260896e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61272462 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601807 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136581769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944éã´é'áãO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6057790778880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426235675981824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6058447141376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚5717.998719557 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6058447141376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105027486 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426235675981824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34516216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6356329098240 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚409188164187136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚420177885202944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95789116 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6036149060674 $ _timestamp‚1.788422218260896e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426236350347264 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105027488 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚409188181226496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241609255 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241629295 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61272462 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚465275845859840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465275664587264 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136601807 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136581769 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚458919359429120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415224330287170 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚420177885202944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136581769 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426236350347264 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚420177903205888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚592 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61272460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458919335489024 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136601807 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6035494622918 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415223658810054 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844Ðß+2â- _wandbruntime‚5718  _runtime‚5718½QÂï&ãéM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5718.730958567 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600223 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415279379375578 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426293522827264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚593 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6059266582528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241641725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6036965495846 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136620287 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 trainer/global_step‚7249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884222189928207e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458980986226688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚420234256244736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34520644 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430°:Õ­öäîM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚426292848448512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocat³„þ%ed‚ 6357352033792 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105041504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136620287 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409243068317696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136600223 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚415279379375578 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058610206720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚415280050852902 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426293522827264 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61280630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚420234256244736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚593 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6059266582528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241641725 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058610206720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚458981010166784 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6036965495846 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136620287 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95801714 trainer/global_step‚7249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105041504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34521084 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884222189928207e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚458980986226688 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚420234256244736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426293522827264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95801272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105041502 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34520644 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241661791 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚420234238241792 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241661791 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136600223 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚465338362200576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6059266582528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _runtime‚5718.730958567 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6357182131712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61280628 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6036311057882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105041502 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241641725 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409243085357056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 trainer/global_step‚7253 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884222724262788e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚422264321808384 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6378745542144 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚422264321808384 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚422264339811328 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚428344788464640 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 137034830 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928J[T@ðËT TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7884222724262788e+09 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 105627062 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚422264321808384 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 6378745542144 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 6057571833338 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚422264321808384 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 137034830 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚422264339811328 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 105627060 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚428344788464640 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 137034830 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 242661890 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242661890 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚461083524059136 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6079955823104 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚61631996 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 bNDeviceStatsMonitor.on_validation_be]£Patch_start/active_bytes.large_pool.allocated‚422264339811328 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚417250158841938 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 105627060 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚428344277631488 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 117441024 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 _runtime‚5772.164282797 _step‚595 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚411192096094208 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 trainer/global_step‚7253 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚428344788464640Óf4qü2ñ- _wandbruntime‚5773  _runtime‚5773|gsôV)òÐR ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 242679998 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚154 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚96404272 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚34765772 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242679998 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 137042132 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 137042132 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 117441024 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6080813779968 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6057935442746 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 105637868 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 105637866 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚467500486521344 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 9643699025 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34765924 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6379110668800 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚96404426 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚428381472786944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚428381983620096 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7253 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884222733907297e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 137058109 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6080320949760 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚61638502 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461121375852544 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 _step‚596 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚422301151837184 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 105637868 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚422301169840128 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚96404426 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚417285816210834 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚61638500 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚428381472786944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚428381983620096 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7253 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884222733907297e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 137058109 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 9955586048 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚467500627902464 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6080320949760 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚61638502 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚461121375852544 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 _step‚596 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚422301151837184 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 141381120 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 B&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918649792671204 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9867258071899414 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013083030469715595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5408139228820801 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3526197671890259 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7732932567596436 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4171716868877411 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2952052652835846 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 epoch‚5 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0913289412856102 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5579743385314941 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5394148826599121 *jet/accuracy_2_of_4‚0.3624871075153351 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3996427655220032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6392867565155029 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.47801461815834045 particle/accuracy_1_of_4‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5256109833717346 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2356976419687271 +particle/specificity‚0.9491503834724426 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4122806787490845 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5809352993965149 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.455987811088562 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916885495185852 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12937672436237335 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15898844599723816 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28288108110427856 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5622450709342957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9825505018234253 particle/accuracy_1_of_1‚1 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10070383548736572 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5905593037605286 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 particle/accuracy_2_of_6‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08694594353437424 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03173007816076279 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4610135853290558 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5730793476104736 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1620999127626419 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2510274052619934 )loss/val_total_loss‚44.30176930794563 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36772486567497253 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6069139242172241 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.541008472442627 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 *jet/accuracy_1_of_6‚0.9807692170143127 jet/accuracy_3_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5321192741394043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10557515174150467 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12514078617095947 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5233237743377686 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9861178994178772 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6152134537696838 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3562518358230591 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48461881279945374 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17867685854434967 +jet/accuracy_3_of_6‚0.07692307978868484 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5397371649742126 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6107068657875061 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018076129257678986 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_5_of_6‚0.01923076994717121 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37320661544799805 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1573026478290558 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5521378517150879 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5432365536689758 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003647587029263377 particle/accuracy_4_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 particle/accuracy_6_of_7‚1 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.98203045129776 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5433892607688904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23023498058319092 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09957413375377655 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0063492064364254475 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13428868353366852 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5556699633598328 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5628951787948608 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_4‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20370371639728546 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5624052286148071 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49784478545188904 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19120897352695465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871904850006104 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4556344747543335 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8788338303565979 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5232218503952026 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26415374875068665 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10444444417953491 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8195986747741699 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26944446563720703 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 _step‚597 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5284171104431152 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.481305867433548 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5346018075942993 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5443140864372253 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5126045346260071 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3395833373069763 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44684097170829773 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1067802682518959 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1171233206987381 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_3_of_5‚0.09367639571428299 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0950726643204689 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_3_of_4‚0.062412139028310776 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894484281539917 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5624052286148071 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 trainer/global_step‚7253 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966248869895935 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8427166938781738 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34583333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023540709167718887 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12728385627269745 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39147287607192993 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12394638359546661 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09146195650100708 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12852463126182556 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.544202983379364 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904633164405823 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15833333134651184 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9511521458625793 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500596046448 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20090089738368988 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4523965120315552 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5551905035972595 particle/accuracy_4_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1844988316297531 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3720720410346985 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5349959135055542 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5499002933502197 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46005746722221375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021028355695307255 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24027560651302338 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919846653938293 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416716337204 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13275344669818878 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19948920607566833 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31634029746055603 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4524397850036621 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12913967669010162 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 (particle/f_score‚0.34681230783462524 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6574074625968933 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6666667461395264 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.02111278846859932 *jet/accuracy_1_of_4‚0.9246519804000854 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17577457427978516 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5992063879966736 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5267310738563538 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05864197760820389 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07183907926082611 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9740399718284607 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005142200738191605 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5981068015098572 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07289386540651321 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5066487193107605 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3765624463558197 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1111111044883728 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15978959202766418 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247461795807 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04480224847793579 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5857478380203247 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 % _timestamp‚1.7884222869174974e+09 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98830646276474 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.980434238910675 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5380872488021851 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9913322925567627 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012800912372767925 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5029857158660889 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5668914318084717 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -jet/accuracy_4_of_4‚0.0038139328826218843 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6741323471069336 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4420718252658844 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4068230390548706 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.584331750869751 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0027855588123202324 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3092593252658844 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35721081495285034 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.08891561627388 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13678161799907684 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5689060688018799 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.164777472615242 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 7 r¹À Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11812865734100342 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41295313835144043 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10726106911897659 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9810739755630493 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23151446878910065 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5662916302680969 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957628846168518 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03764134272933006µj72{øÔ¦ ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5408139228820801 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3526197671890259 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7732932567596436 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4171716868877411 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2952052652835846 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.01923076994717121 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.0555555559694767 epoch‚5 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0913289412856102 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5579743385314941 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5394148826599121 *jet/accuracy_2_of_4‚0.3624871075153351 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3996427655220032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6392867565155029 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.47801461815834045 particle/accuracy_1_of_4‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5256109833717346 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.2356976419687271 +particle/specificity‚0.9491503834724426 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4122806787490845 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859649300575256 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5809352993965149 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.455987811088562 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916885495185852 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12937672436237335 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15898844599723816 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28288108110427856 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5622450709342957 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9825505018234253 particle/accuracy_1_of_1‚1 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5627976059913635 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10070383548736572 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_3_of_4‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5905593037605286 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833432674408 particle/accuracy_2_of_6‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08694594353437424 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03173007816076279 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4610135853290558 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5730793476104736 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1620999127626419 ;"Purity/0FRt*SRqqt3FBt/event_purity‚0.061403512954711914 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2510274052619934 )loss/val_total_loss‚44.30176930794563 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36772486567497253 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6069139242172241 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.541008472442627 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 *jet/accuracy_1_of_6‚0.9807692170143127 jet/accuracy_3_of_7‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5321192741394043 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.10557515174150467 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12514078617095947 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5233237743377686 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9861178994178772 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6152134537696838 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3562518358230591 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.48461881279945374 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17867685854434967 +jet/accuracy_3_of_6‚0.07692307978868484 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5397371649742126 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6107068657875061 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018076129257678986 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 +jet/accuracy_5_of_6‚0.01923076994717121 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37320661544799805 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1573026478290558 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5521378517150879 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5432365536689758 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.003647587029263377 particle/accuracy_4_of_6‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 particle/accuracy_6_of_7‚1 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 5 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.98203045129776 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5433892607688904 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23023498058319092 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09957413375377655 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0063492064364254475 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13428868353366852 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5556699633598328 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5628951787948608 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_4_of_4‚1 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.215277761220932 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20370371639728546 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5624052286148071 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49784478545188904 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19120897352695465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9871904850006104 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4556344747543335 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8788338303565979 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5232218503952026 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26415374875068665 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10444444417953491 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7777777910232544 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8195986747741699 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.26944446563720703 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 _step‚597 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5284171104431152 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.481305867433548 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5346018075942993 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5443140864372253 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5126045346260071 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424242198467255 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3395833373069763 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.44684097170829773 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1067802682518959 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1171233206987381 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 +jet/accuracy_3_of_5‚0.09367639571428299 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0950726643204689 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_3_of_4‚0.062412139028310776 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9894484281539917 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5624052286148071 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 trainer/global_step‚7253 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966248869895935 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 particle/accuracy_1_of_3‚1 (particle/accuracy‚0.8427166938781738 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34583333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023540709167718887 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12728385627269745 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.39147287607192993 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.12394638359546661 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09146195650100708 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12852463126182556 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.544202983379364 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5402299165725708 particle/accuracy_4_of_5‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904633164405823 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.15833333134651184 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9511521458625793 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500596046448 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.20090089738368988 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4523965120315552 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09574468433856964 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5551905035972595 particle/accuracy_4_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1844988316297531 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3720720410346985 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5349959135055542 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5499002933502197 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.46005746722221375 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0021028355695307255 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24027560651302338 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919846653938293 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.0885416716337204 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13275344669818878 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19948920607566833 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31634029746055603 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4524397850036621 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12913967669010162 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 (particle/f_score‚0.34681230783462524 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6574074625968933 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.6666667461395264 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.02111278846859932 *jet/accuracy_1_of_4‚0.9246519804000854 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17577457427978516 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5992063879966736 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5267310738563538 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.05864197760820389 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.02083333395421505 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.07183907926082611 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9740399718284607 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998635470867157 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005142200738191605 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5981068015098572 8 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.02083333395421505 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07289386540651321 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184397459030151 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5066487193107605 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3765624463558197 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1111111044883728 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15978959202766418 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247461795807 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04480224847793579 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5857478380203247 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 % _timestamp‚1.7884222869174974e+09 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.98830646276474 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.980434238910675 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5380872488021851 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9913322925567627 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012800912372767925 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5029857158660889 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5668914318084717 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -jet/accuracy_4_of_4‚0.0038139328826218843 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6741323471069336 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4420718252658844 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4068230390548706 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.584331750869751 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0027855588123202324 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3092593252658844 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35721081495285034 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.08891561627388 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.13678161799907684 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5689060688018799 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.164777472615242 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03095238097012043 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3636363744735718 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_p«7ö()urity‚0.9166666865348816 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.11812865734100342 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5714285969734192 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41295313835144043 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03448275849223137 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10726106911897659 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9810739755630493 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.23151446878910065 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5662916302680969 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9957628846168518 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03764134272933006 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12405510991811752 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5744110345840454 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_7_of_7‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2000928372144699 particle/accuracy_1_of_6‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13434472680091858 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.44355031847953796 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1918639838695526 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2566879391670227 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,particle/sensitivity‚0.26758795976638794 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5192307829856873 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17384247481822968 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11519645154476166 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20489969849586487 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15978959202766418 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.007426303345710039 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.015456990338861942 jet/accuracy_6_of_6‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34995099902153015 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6072916388511658 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.014965986832976341 *jet/accuracy_1_of_1‚0.5109090805053711 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2946931719779968 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2558228075504303 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13544641435146332 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.256261944770813 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 6 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.761904776096344 _runtime‚5786.657476887 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9736111164093018 5validation_average_jet_accuracy‚0.448695570230484 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976710081100464 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5174904465675354 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08288352191448212 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918649792671204 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9867258071899414 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013083030469715595 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297Õ “™ù‚(:… ’ÑäÔð »~ gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.716' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9432 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qV"7qú‚(:æ ’ÑäÔ®¨¼ network.recv‚ 2.136221e+06 disk.sda1.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚24385.98583984375 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.671875 cpu‚25.36363707572378 network.sent‚ 1.7031534e+07 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.rbd0.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚24385.98583984375 disk.rbd1.in‚0% disk./.usageGB‚259.03357696533203 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda.in‚ 126.77734375# proc.memory.rssMB‚ 3242.09765625* disk./.usagePercent‚17.383960693791067 disk.nvme0c0n1.in‚4.671875 disk.sda3.out‚350.0625 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1431.12109375 proc.cpu.threads‚17 disk.nvme0n1.in‚4.671875 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚350.0625* proc.memory.percent‚19.788193702697754 disk.rbd0.in‚0( disk.nvme0c0n1.out‚24385.98583984375 disk.drbd2265.in‚0 disk.sda.out‚350.0625$ memory_percent‚91.26513004302979Š+š’û‚(:‡ ¡ÑäÔÀªþ} gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚163.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ïÝmü‚(:â ¡ÑäÔôŸ# disk.dm-0.out‚24902.90380859375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda2.out‚0* disk./.usagePercent‚17.384908433682824& disk.nvme0n1.out‚24902.90380859375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚24902.90380859375( proc.memory.availableMB‚ 1901.9453125 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.nvme0n1.in‚4.671875 disk.sda2.in‚0 network.sent‚ 1.7155524e+07 disk.dm-1.out‚350.5625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875$ disk./.usageGB‚259.0476989746094 disk.dm-1.in‚ 126.77734375 disk.sda3.out‚350.5625 proc.cpu.threads‚17 disk.rbd1.out‚ 3.06640625 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.921329498291016 network.recv‚ 2.142293e+06 disk.drbd2080.in‚0 disk.sda.out‚350.5625 disk.drbd2265.in‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2772.390625 cpu‚25.36180340683056 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ memory_percent‚88.39144706726074ùi¡’ý‚(:‡ °ÑäÔݾ~ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—Dgórþ‚(:ç °ÑäÔ²ç‚$ disk./.usageGB‚259.0568618774414 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 351.05078125 disk.drbd1716.out‚0$ disk.nvme0n1.out‚25413.451171875 disk.rbd1.in‚0! proc.memory.rssMB‚ 2772.390625 network.recv‚ 2.147274e+06 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0! disk.dm-0.out‚25413.451171875( proc.memory.availableMB‚ 1901.9453125 proc.cpu.threads‚17 disk.sda1.in‚0& disk.nvme0c0n1.out‚25413.451171875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 351.05078125 disk.rbd1.out‚ 3.06640625$ memory_percent‚88.39144706726074 network.sent‚ 1.7161915e+07) disk./.usagePercent‚17.38552336378007* proc.memory.percent‚16.921329498291016 cpu‚25.36096261961874 disk.rbd0.out‚0 disk.sda3.out‚ 351.05078125 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0¿Ö´2ÿ- _wandbruntime‚5826  _runtime‚5826#éYÅ€¿ % lr-AdamW/pg1‚6.86193246145767e-05 % lr-AdamW/pg2‚6.86193246145767e-05 trainer/global_step‚7299 % _timestamp‚1.7884223269206617e+09 _runtime‚5826.658273897 _step‚598‰íÐ,ÊÄ % lr-AdamW/pg2‚6.86193246145767e-05 trainer/global_step‚7299 % _timestamp‚1.7884223269206617e+09 _runtime‚5826.658273897 _step‚598 % lr-AdamW/pg1‚6.86193246145767e-05Ö g•ñ2‚- _wandbruntime‚5826  _runtime‚5826¦›ïå'ƒßO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106409064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6422821193728 trainer/global_step‚7299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425315828959744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97118731 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420256803502404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6099908900164 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6099254462408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106409064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244378848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚5826.659252433 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106409066 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6121796554752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625514496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431438299879936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137989822 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470810699055616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35028117 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414156877562880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464388067523072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106409066 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62090612 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137969784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884223269214516e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6122452917248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97118303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62090614 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422655472640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944ù§ê'„äO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244378848 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚5826.659252433 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106409066 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6121796554752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431437625514496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431438299879936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137989822 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚425315846962688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚470810699055616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35028117 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414156877562880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431437625514496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚464388067523072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425315846962688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106409066 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚428 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62090612 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137969784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884223269214516e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6122452917248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244398888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420256132025288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97118303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62090614 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6422655472640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414156894602240 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106409064 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6422821193728 trainer/global_step‚7299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425315828959744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚464388043582976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97118731 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420256803502404 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6099908900164 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137989822 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6099254462408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106409064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430×!­Ÿù2…- _wandbruntime‚5827  _runtime‚5827ö'03v†çM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988238 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6122615982592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6122615982592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6423688618496 C=DeviceStatsMonitor.on_train_batch_¦Å-Æw#end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425372181998592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138008302 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35032180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100070897372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚7299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423539687936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 # _timestamp‚1.78842232766355e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131380 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97130960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431494797981184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272358400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5827.401571797fÂ#ò&‡ìM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35032180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106423080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62098782 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6100070897372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚7299 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6423539687936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 # _timestamp‚1.78842232766355e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6123272358400 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚464449718260736 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35032598 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420311852590812 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425372200001536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97131380 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420312524068136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414211781693440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97130960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 172870656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431494797981184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6123272358400 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244431384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚425372200001536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 _runtime‚5827.401571797 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137988238 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6122615982592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6100725335336 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244411318 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6122615982592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6423688618496 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425372181998592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138008302 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088Ø£YÓ‡2ˆ- _wandbruntime‚5827  _runtime‚5827&³PZ‰Ô _step‚601 1loss/FBt4/assignment_loss‚0.11959410985784263 .loss/FRt3/detection_loss‚2.125226806776196 0loss/FRt2/assignment_loss‚3.4053615413863083 1loss/FBt1/assignment_loss‚0.34017179876432885 1loss/FBt2/assignment_loss‚0.12541913706809282 % _timestamp‚1.7884223276652336e+09 0loss/FRt4/assignment_loss‚1.3669161025820107 /loss/FBt2/detection_loss‚3.2510327781949724 1loss/SRqqt2/assignment_loss‚1.575913509560956 0loss/SRqqt2/detection_loss‚2.439961075782776 1loss/SRqqt3/detection_loss‚1.9577825283631682 0loss/SRqqt4/detection_loss‚0.908241411397124 1loss/FBt3/assignment_loss‚0.09206688767299057 .loss/FBt3/detection_loss‚2.575499081611633 %loss/total_loss‚40.36708147524843 trainer/global_step‚7299 /loss/FRt1/assignment_loss‚2.912095107462095 /loss/FRt3/assignment_loss‚2.528207331311469 2loss/SRqqt1/assignment_loss‚1.1298454282235126 2loss/SRqqt3/assignment_loss‚1.5083078426153709 1loss/SRqqt4/assignment_loss‚1.083773310707115 /loss/FRt1/detection_loss‚2.1847158903660984 /loss/FRt4/detection_loss‚1.4696636907756329 epoch‚6 /loss/FRt2/detection_loss‚2.9278473545765054 1loss/SRqqt1/detection_loss‚1.6558744192731625 .loss/FBt1/detection_loss‚2.039390715105193 /loss/FBt4/detection_loss‚0.6441736158138762 _runtime‚5827.402676372WËðJ_ŠÙ % _timestamp‚1.7884223276652336e+09 0loss/FRt4/assignment_loss‚1.3669161025820107 /loss/FBt2/detection_loss‚3.2510327781949724 1loss/SRqqt2/assignment_loss‚1.575913509560956 0loss/SRqqt2/detection_loss‚2.439961075782776 1loss/SRqqt3/detection_loss‚1.9577825283631682 0loss/SRqqt4/detection_loss‚0.908241411397124 1loss/FBt3/assignment_loss‚0.09206688767299057 .loss/FBt3/detection_loss‚2.575499081611633 %loss/total_loss‚40.36708147524843 trainer/global_step‚7299 /loss/FRt1/assignment_loss‚2.912095107462095 /loss/FRt3/assignment_loss‚2.528207331311469 2loss/SRqqt1/assignment_loss‚1.1298454282235126 2loss/SRqqt3/assignment_loss‚1.5083078426153709 1loss/SRqqt4/assignment_loss‚1.083773310707115 /loss/FRt1/detection_loss‚2.1847158903660984 /loss/FRt4/detection_loss‚1.4696636907756329 epoch‚6 /loss/FRt2/detection_loss‚2.9278473545765054 1loss/SRqqt1/detection_loss‚1.6558744192731625 .loss/FBt1/detection_loss‚2.039390715105193 /loss/FBt4/detection_loss‚0.6441736158138762 _runtime‚5827.402676372 _step‚601 1loss/FBt4/assignment_loss‚0.11959410985784263 .loss/FRt3/detection_loss‚2.125226806776196 0loss/FRt2/assignment_loss‚3.4053615413863083 1loss/FBt1/assignment_loss‚0.34017179876432885 1loss/FBt2/assignment_loss‚0.12541913706809282Ù7W¨Ù’‹‚(:‡ ¿ÑäÔÇÞ¯~ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð[JpŒ‚(:å ¿ÑäÔ™û½€! proc.memory.rssMB‚ 2772.390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875 disk.dm-1.out‚351.5625 disk.sda1.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚259.07445907592773 disk.sda.out‚351.5625$ memory_percent‚88.39013576507568 disk.drbd2080.out‚0# disk.dm-0.out‚25454.44384765625 disk.rbd1.in‚0 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 network.recv‚ 2.150653e+06 disk.drbd2265.in‚0 disk.drbd2266.in‚0& disk.nvme0n1.out‚25454.44384765625( disk.nvme0c0n1.out‚25454.44384765625 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda1.out‚0 disk.sda3.out‚351.5625 network.sent‚ 1.7167325e+07 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda2.out‚0 cpu‚25.360128884922204* disk./.usagePercent‚17.386704326535522* proc.memory.percent‚16.921329498291016 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1902.16015625˜Æ$|œ‚(:‘ ÎÑäÔòŒÆ~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.884' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.17679999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý0Œ‘fŽ‚(:Û ÎÑäÔ¹Åé€ disk.rbd1.out‚ 3.06640625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0$ memory_percent‚88.39006423950195% disk.nvme0n1.out‚25498.2939453125 disk.sda3.out‚352.125 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875' disk.nvme0c0n1.out‚25498.2939453125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 network.sent‚ 1.7275981e+07 network.recv‚ 2.156873e+06 disk.rbd1.in‚0 disk.sda.out‚352.125 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875" disk.dm-0.out‚25498.2939453125 disk.drbd2266.out‚0) disk./.usagePercent‚17.38793162665467 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375' proc.memory.availableMB‚ 1902.171875* proc.memory.percent‚16.921329498291016% disk./.usageGB‚259.09274673461914 disk.sda2.out‚0 disk.dm-1.out‚352.125 cpu‚25.35933417680904 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2772.390625Iº&’‚(:‡ ÝÑäÔ…”þ} gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.082' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.2164 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°¹tñu‚(:ê ÝÑäÔ„²ˆ€* disk./.usagePercent‚17.388796420105333 disk.sda2.out‚0 disk.dm-1.out‚ 352.8671875 disk.rbd1.in‚0 disk.sda3.out‚ 352.8671875 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17 disk.sda1.out‚0! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.7281801e+07$ disk./.usageGB‚259.1056327819824 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.drbd2265.out‚0$ memory_percent‚88.39013576507568 cpu‚25.358553353366332% disk.nvme0n1.out‚25532.8291015625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875" disk.dm-0.out‚25532.8291015625) proc.memory.availableMB‚ 1902.16015625 disk.sda.out‚ 352.8671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921329498291016 network.recv‚ 2.160968e+06 disk.sda.in‚ 126.77734375' disk.nvme0c0n1.out‚25532.8291015625 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375̆c2‘- _wandbruntime‚5865  _runtime‚58654ôÄ’¾ % lr-AdamW/pg1‚6.74247590317537e-05 % lr-AdamW/pg2‚6.74247590317537e-05 trainer/global_step‚7349 $ _timestamp‚1.788422366195277e+09 _runtime‚5865.932905831 _step‚602¢¥}ÌÉ“Ã % lr-AdamW/pg1‚6.74247590317537e-05 % lr-AdamW/pg2‚6.74247590317537e-05 trainer/global_step‚7349 $ _timestamp‚1.788422366195277e+09 _runtime‚5865.932905831 _step‚602ÚPˆ-¼2”- _wandbruntime‚5865  _runtime‚5865‰·¢@•ßO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434296263262720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5865.933771284 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423042173892696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97753256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025740 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473936357963264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872ëÙ1Šê'–äO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97753256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚467470604406272 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 246025740 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473936357963264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107109864 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚434296263262720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚5865.933771284 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚434296937628160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚434296263262720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428133480902144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚423042173892696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128ÛTãÀ2—- _wandbruntime‚5866  _runtime‚58667ÙÜEí&˜çM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428189851943936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6466464915968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434354110108160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998696119808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601788416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35258656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97765392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚604 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258164224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6164258164224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998877379072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934354 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466622235136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423098565935544 _runtime‚5866.716277497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 # _timestamp‚1.78842236697809e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088àâ®—/™ìM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107123880 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138934354 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6466622235136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚423098565935544 _runtime‚5866.716277497 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 # _timestamp‚1.78842236697809e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428189851943936 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416957005260800 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416956988221440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428189851943936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6466464915968 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚434354110108160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚428189833940992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246038170 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107123882 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428189833940992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473998696119808 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6163601788416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35258656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107123880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97765392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚604 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107123882 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246058236 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6164258164224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246058236 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6164258164224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚434353435729408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473998877379072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138934354ÜÖX¾2š- _wandbruntime‚5866  _runtime‚5866lÅÌmX›Ò 0loss/SRqqt2/detection_loss‚2.776965494950612 /loss/FRt2/assignment_loss‚3.551066633817312 0loss/SRqqt3/detection_loss‚1.724148442252324 0loss/FRt4/assignment_loss‚1.4296282256642978 2loss/SRqqt3/assignment_loss‚1.4613676754614482 /loss/FRt4/detection_loss‚1.9109171714323263 _runtime‚5866.717684392 0loss/FRt1/assignment_loss‚3.2308380007743835 1loss/FBt2/assignment_loss‚0.45147896534763277 1loss/SRqqt1/detection_loss‚1.7588634266247671 1loss/SRqqt4/detection_loss‚0.8925945239796026 /loss/FBt2/detection_loss‚2.2202221676707268 .loss/FBt4/detection_loss‚0.657129092896475 epoch‚6 trainer/global_step‚7349 2loss/SRqqt1/assignment_loss‚1.3091694341912385 2loss/SRqqt4/assignment_loss‚0.7172650617320795 %loss/total_loss‚42.45264824938688 % _timestamp‚1.7884223669801745e+09 /loss/FRt3/assignment_loss‚2.392751132900065 /loss/FRt2/detection_loss‚2.7928324575359755 .loss/FBt3/detection_loss‚2.811450791358948 0loss/FBt3/assignment_loss‚1.0078851241618394 0loss/FBt4/assignment_loss‚0.1512087099133358 /loss/FRt3/detection_loss‚2.3529872958633034 _step‚605 1loss/SRqqt2/assignment_loss‚1.829620161652565 1loss/FBt1/assignment_loss‚0.09243948997131415 .loss/FRt1/detection_loss‚2.124960765890453 .loss/FBt1/detection_loss‚2.804858003343855²²¡‚]œ× 1loss/SRqqt2/assignment_loss‚1.829620161652565 1loss/FBt1/assignment_loss‚0.09243948997131415 .loss/FRt1/detection_loss‚2.124960765890453 .loss/FBt1/detection_loss‚2.804858003343855 0loss/SRqqt2/detection_loss‚2.776965494950612 /loss/FRt2/assignment_loss‚3.551066633817312 0loss/SRqqt3/detection_loss‚1.724148442252324 0loss/FRt4/assignment_loss‚1.4296282256642978 2loss/SRqqt3/assignment_loss‚1.4613676754614482 /loss/FRt4/detection_loss‚1.9109171714323263 _runtime‚5866.717684392 0loss/FRt1/assignment_loss‚3.2308380007743835 1loss/FBt2/assignment_loss‚0.45147896534763277 1loss/SRqqt1/detection_loss‚1.7588634266247671 1loss/SRqqt4/detection_loss‚0.8925945239796026 /loss/FBt2/detection_loss‚2.2202221676707268 .loss/FBt4/detection_loss‚0.657129092896475 epoch‚6 trainer/global_step‚7349 2loss/SRqqt1/assignment_loss‚1.3091694341912385 2loss/SRqqt4/assignment_loss‚0.7172650617320795 %loss/total_loss‚42.45264824938688 % _timestamp‚1.7884223669801745e+09 /loss/FRt3/assignment_loss‚2.392751132900065 /loss/FRt2/detection_loss‚2.7928324575359755 .loss/FBt3/detection_loss‚2.811450791358948 0loss/FBt3/assignment_loss‚1.0078851241618394 0loss/FBt4/assignment_loss‚0.1512087099133358 /loss/FRt3/detection_loss‚2.3529872958633034 _step‚605ÝxE„œ‚(:‘ ìÑäÔÕù´~ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.372' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.27440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯õ<@[ž‚(:Ð ìÑäÔƒ¡Ú€ disk.sda2.out‚0 disk.sda3.out‚353.25" disk.dm-0.out‚25582.2021484375 disk.drbd2266.in‚0 disk.sda.out‚353.25 cpu‚25.357806325710104 network.sent‚ 1.7288135e+07 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.671875 disk.rbd1.out‚ 3.06640625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚353.25 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 network.recv‚ 2.165429e+06 disk./.usageGB‚259.125 disk.drbd1716.in‚0% disk.nvme0n1.out‚25582.2021484375 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚88.39051723480225! proc.memory.rssMB‚ 2772.390625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.671875' disk.nvme0c0n1.out‚25582.2021484375 disk.drbd2080.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1902.09765625* proc.memory.percent‚16.921329498291016 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375) disk./.usagePercent‚17.39009617035667Ù1Öæ’Ÿ‚(:‡ ûÑäÔŽ’¡~ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Móm ‚(:â ûÑäÔ¼Ò– network.recv‚ 2.170958e+06 disk.sda.out‚ 353.84375 disk.dm-0.in‚4.671875 cpu‚25.357022385683955" disk.dm-0.out‚25618.3330078125 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 353.84375 proc.cpu.threads‚17$ disk./.usageGB‚259.1412048339844 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚25618.3330078125 disk.dm-1.out‚ 353.84375# memory_percent‚88.3920431137085! proc.memory.rssMB‚ 2772.390625 network.sent‚ 1.739607e+07 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1901.84765625 disk.sda1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚25618.3330078125 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0* proc.memory.percent‚16.921329498291016* disk./.usagePercent‚17.391183690362123 disk.rbd1.out‚ 3.06640625 disk.rbd0.in‚0½ßº§2¡- _wandbruntime‚5905  _runtime‚5905žC¦Ç¢Á & lr-AdamW/pg1‚6.623371924821695e-05 & lr-AdamW/pg2‚6.623371924821695e-05 trainer/global_step‚7399 % _timestamp‚1.7884224058284323e+09 _runtime‚5905.566065756 _step‚606öS<éÌ£Æ & lr-AdamW/pg1‚6.623371924821695e-05 & lr-AdamW/pg2‚6.623371924821695e-05 trainer/global_step‚7399 % _timestamp‚1.7884224058284323e+09 _runtime‚5905.566065756 _step‚606Þ!Fx2¤- _wandbruntime‚5905  _runtime‚5905üÞfå'¥ßO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚430951150847488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62907414 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470553117349376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6508419012608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35479830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139841926 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6180925141224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437154901010944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚470553117349376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247652592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6508419012608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437154901010944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6204424528896 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35479830 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139821888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139841926 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425828887237220 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6180925141224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ß%-(2§- _wandbruntime‚5906  _runtime‚5906ȯ Ïø¨éM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509440007680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425883936325628 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98399511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470614792027136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884224065714576e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437212747856384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419702194749440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35483931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124038192640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860406 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576188 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35484382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124232034816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944ƒˆŒô&©îM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431007485883392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139840342 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚419702211788800 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247685088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6182396014152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62915582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247685088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35483931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124038192640 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139860406 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6181741576188 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247665022 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35484382 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477124232034816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6509440007680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437212747856384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431007485883392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚7399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425883936325628 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107824680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107824680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98399511 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚470614792027136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884224065714576e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437212747856384 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431007503886336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6205243970048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚419702194749440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431007503886336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6204587594240 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140767978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538759168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99023699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551516053504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433768784786944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918388 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756œ¾,qé'¸ãO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249259404 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140747940 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108511466 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433768802789888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35707887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473635678172672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚422392514186240 trainer/global_step‚7449 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚440014213124608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473635654232576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6551352429568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚422392497146880 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249279444 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140767978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433768784786944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚440013538759168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244753972224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99023699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6551516053504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433768784786944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222414918388 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140747940 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚440013538759168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140767978 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221760480632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433768802789888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚428614257627512 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63315814 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63315812 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552207536640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5945.661610044 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99036289 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249702154752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884224459226594e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573400064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440071385604608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140766394 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99036729 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414gˆ_ô&»îM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433825155828736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433825137825792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140766394 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99036729 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223231353560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222576915596 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249311940 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6246229775872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108525480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 trainer/global_step‚7449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6552207536640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚5945.661610044 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99036289 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480249702154752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚428670649670360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63323982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108525482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249311940 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249291874 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884224459226594e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246229775872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249291874 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433825155828736 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245573400064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚440071385604608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0äý6Ü72¼- _wandbruntime‚5945  _runtime‚5945ä—ä`P½Ó 0loss/SRqqt3/detection_loss‚1.712637062235312 /loss/FBt3/detection_loss‚2.0073929242789745 /loss/FRt1/assignment_loss‚3.129531505260062 /loss/FRt2/assignment_loss‚4.050516637166341 2loss/SRqqt3/assignment_loss‚1.6348588246513496 0loss/FBt2/assignment_loss‚0.3655564930231696 /loss/FRt4/detection_loss‚2.1410921942442656 epoch‚6 trainer/global_step‚7449 _runtime‚5945.662372002 1loss/SRqqt4/detection_loss‚1.0439203100708814 /loss/FBt1/detection_loss‚2.7153464257717133 /loss/FBt4/detection_loss‚0.6351756167939554 1loss/SRqqt4/assignment_loss‚0.931868244177447 $ _timestamp‚1.788422445924844e+09 2loss/SRqqt1/assignment_loss‚1.3825871946244706 0loss/FBt1/assignment_loss‚0.3066644402486937 1loss/FBt4/assignment_loss‚0.10131384639370963 /loss/FRt1/detection_loss‚ù3rv 1.9959770124009315 /loss/FRt2/detection_loss‚2.9147409637769064 _step‚613 /loss/FRt3/assignment_loss‚2.393748830258846 0loss/FRt4/assignment_loss‚1.4079330197789452 1loss/SRqqt2/detection_loss‚2.2553956315324113 $loss/total_loss‚42.9313465377832 2loss/SRqqt2/assignment_loss‚1.8226481923783147 1loss/FBt3/assignment_loss‚0.16768484317502796 /loss/FRt3/detection_loss‚2.6100501298904417 1loss/SRqqt1/detection_loss‚1.6522547042951352 .loss/FBt2/detection_loss‚3.552451491355896u@oØ^¾Ø 1loss/SRqqt4/assignment_loss‚0.931868244177447 $ _timestamp‚1.788422445924844e+09 2loss/SRqqt1/assignment_loss‚1.3825871946244706 0loss/FBt1/assignment_loss‚0.3066644402486937 1loss/FBt4/assignment_loss‚0.10131384639370963 /loss/FRt1/detection_loss‚1.9959770124009315 /loss/FRt2/detection_loss‚2.9147409637769064 _step‚613 /loss/FRt3/assignment_loss‚2.393748830258846 0loss/FRt4/assignment_loss‚1.4079330197789452 1loss/SRqqt2/detection_loss‚2.2553956315324113 $loss/total_loss‚42.9313465377832 2loss/SRqqt2/assignment_loss‚1.8226481923783147 1loss/FBt3/assignment_loss‚0.16768484317502796 /loss/FRt3/detection_loss‚2.6100501298904417 1loss/SRqqt1/detection_loss‚1.6522547042951352 .loss/FBt2/detection_loss‚3.552451491355896 0loss/SRqqt3/detection_loss‚1.712637062235312 /loss/FBt3/detection_loss‚2.0073929242789745 /loss/FRt1/assignment_loss‚3.129531505260062 /loss/FRt2/assignment_loss‚4.050516637166341 2loss/SRqqt3/assignment_loss‚1.6348588246513496 0loss/FBt2/assignment_loss‚0.3655564930231696 /loss/FRt4/detection_loss‚2.1410921942442656 epoch‚6 trainer/global_step‚7449 _runtime‚5945.662372002 1loss/SRqqt4/detection_loss‚1.0439203100708814 /loss/FBt1/detection_loss‚2.7153464257717133 /loss/FBt4/detection_loss‚0.6351756167939554å$±$_¿‚(:’ ·ÒäÔ™š° gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.296' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.459199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{é'qÀ‚(:æ ·ÒäÔ–»â€* proc.memory.percent‚16.921353340148926 proc.cpu.threads‚17 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚25776.16845703125 disk.sda2.in‚0 cpu‚25.35390420937637& disk.nvme0n1.out‚25776.16845703125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 356.08984375! memory_percent‚88.39111328125 proc.memory.availableMB‚1902 disk.drbd2080.out‚0 disk.sda3.in‚ 126.77734375 network.recv‚ 2.189388e+06 network.sent‚ 1.7522277e+07* disk./.usagePercent‚17.393491854290836# proc.memory.rssMB‚ 2772.39453125 disk.dm-0.in‚4.671875% disk./.usageGB‚259.17559814453125 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.sda3.out‚ 356.08984375 disk.drbd2265.in‚0 disk.sda.out‚ 356.08984375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.rbd0.out‚0# disk.dm-0.out‚25776.16845703125 disk.drbd2266.in‚0šç’Á‚(:‡ ÆÒäÔÑ„¨~ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡‹•›m‚(:â ÆÒäÔ‰‹¸€ cpu‚25.353188582582494 disk.sda2.out‚0 disk.drbd2266.in‚0) disk./.usagePercent‚17.39364136269083 disk.sda1.out‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 2.195544e+06 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875" disk.dm-0.out‚25812.1884765625 disk.dm-1.out‚356.6875) proc.memory.availableMB‚ 1901.99609375# proc.memory.rssMB‚ 2772.39453125 disk.drbd2265.out‚0 network.sent‚ 1.7630813e+07$ disk./.usageGB‚259.1778259277344 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚25812.1884765625 disk.sda3.out‚356.6875* proc.memory.percent‚16.921353340148926 disk.rbd1.in‚0 disk.sda.out‚356.6875 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚25812.1884765625 disk.drbd2266.out‚0$ memory_percent‚88.39113712310791øÊRx2Ã- _wandbruntime‚5984  _runtime‚5984€ë×zÇÄÁ & lr-AdamW/pg1‚6.386320913612691e-05 & lr-AdamW/pg2‚6.386320913612691e-05 trainer/global_step‚7499 % _timestamp‚1.7884224848367915e+09 _runtime‚5984.574320857 _step‚614¿K¿ÌÅÆ % _timestamp‚1.7884224848367915e+09 _runtime‚5984.574320857 _step‚614 & lr-AdamW/pg1‚6.386320913612691e-05 & lr-AdamW/pg2‚6.386320913612691e-05 trainer/global_step‚7499ædSù§2Æ- _wandbruntime‚5984  _runtime‚5984EËå'ÇßO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6286396140544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35936300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141694030 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141694030 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476718191115776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6594305062912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6285739778048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99660069 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141673992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35935857 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109212264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109212266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63724214 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312687937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425137720714240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884224848375244e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425137703674880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442872850872832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63724212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7499 _runtime‚5984.575365689 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436586436729344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Ö ÏÆê'ÈäO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109212264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚483312496178688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431400299494920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6286396140544 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35936300 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431400970972036 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141694030 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚442872850872832 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141694030 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚476718191115776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6263250257796 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚442872176507392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6594305062912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6285739778048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚436586436729344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99660069 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚615 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141673992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35935857 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109212264 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109212266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63724214 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6285739778048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6286396140544 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚483312687937024 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425137720714240 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 % _timestamp‚1.7884224848375244e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚442872176507392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚425137703674880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚442872850872832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63724212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚436586454732288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚7499 _runtime‚5984.575365689 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚436586436729344 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99672778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚7499 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286559205888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes9ˆ(Ñ.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 _runtime‚5985.329138479 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35940398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483375193123328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476779841853440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35940845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286559205888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436642807771136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696ÁÓœô&ÍîM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚442930023352832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250938792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6264066692968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6287215581696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35940398 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6263412255004 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚483375193123328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚476779841853440 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35940845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6286559205888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚436642807771136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6287215581696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425192624844800 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6595327329792 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6595159524864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚436642789768192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99672778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63732380 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚7499 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250938792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109226280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425192607805440 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚436642807771136 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431456691537768 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6286559205888 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109226282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431456020060444 _runtime‚5985.329138479èª5~2Î- _wandbruntime‚5985  _runtime‚5985°•p[ÏÕ /loss/FBt1/detection_loss‚1.8983957717816036 /loss/FBt4/detection_loss‚0.6850903396789605 /loss/FRt4/assignment_loss‚1.137302467163573 /loss/FRt4/detection_loss‚1.9983346236354493 1loss/SRqqt2/detection_loss‚2.8716628643182607 %loss/total_loss‚42.69783228330732 trainer/global_step‚7499 /loss/FBt2/detection_loss‚1.9803449392318726 _step‚617 % _timestamp‚1.7884224855928757e+09 0loss/FRt1/assignment_loss‚3.6821851634210154 /loss/FRt2/assignment_loss‚3.717156646343378 .loss/FRt3/assignment_loss‚2.24539586688791 2loss/SRqqt3/assignment_loss‚1.1858834631741046 1loss/SRqqt1/detection_loss‚2.0479623917923417 /loss/FBt3/detection_loss‚2.8551102757453917 epoch‚6 2loss/SRqqt1/assignment_loss‚1.6178482067446376 /loss/FRt1/detection_loss‚2.4759140245376097 1loss/SRqqt3/detection_loss‚1.5631311166499342 1loss/SRqqt4/detection_loss‚1.3034381866455078 _runtime‚5985.330331551 2loss/SRqqt2/assignment_loss‚1.3781800619684732 2loss/SRqqt4/assignment_loss‚0.9758263161570272 0loss/FBt1/assignment_loss‚0.6167676886543632 0loss/FBt3/assignment_loss‚0.5135153544135391 0loss/FBt4/assignment_loss‚0.1343152932539896 /loss/FRt2/detection_loss‚3.0127238012277164 /loss/FRt3/detection_loss‚2.0148210815553154 0loss/FBt2/assignment_loss‚0.7865263383253478¢ñô`ÐÚ /loss/FBt4/detection_loss‚0.6850903396789605 /loss/FRt4/assignment_loss‚1.137302467163573 /loss/FRt4/detection_loss‚1.9983346236354493 1loss/SRqqt2/detection_loss‚2.8716628643182607 %loss/total_loss‚42.69783228330732 trainer/global_step‚7499 /loss/FBt2/detection_loss‚1.9803449392318726 _step‚617 % _timestamp‚1.7884224855928757e+09 0loss/FRt1/assignment_loss‚3.6821851634210154 /loss/FRt2/assignment_loss‚3.717156646343378 .loss/FRt3/assignment_loss‚2.24539586688791 2loss/SRqqt3/assignment_loss‚1.1858834631741046 1loss/SRqqt1/detection_loss‚2.0479623917923417 /loss/FBt3/detection_loss‚2.8551102757453917 epoch‚6 2loss/SRqqt1/assignment_loss‚1.6178482067446376 /loss/FRt1/detection_loss‚2.4759140245376097 1loss/SRqqt3/detection_loss‚1.5631311166499342 1loss/SRqqt4/detection_loss‚1.3034381866455078 _runtime‚5985.330331551 2loss/SRqqt2/assignment_loss‚1.3781800619684732 2loss/SRqqt4/assignment_loss‚0.9758263161570272 0loss/FBt1/assignment_loss‚0.6167676886543632 0loss/FBt3/assignment_loss‚0.5135153544135391 0loss/FBt4/assignment_loss‚0.1343152932539896 /loss/FRt2/detection_loss‚3.0127238012277164 /loss/FRt3/detection_loss‚2.0148210815553154 0loss/FBt2/assignment_loss‚0.7865263383253478 /loss/FBt1/detection_loss‚1.8983957717816036é³.ù’Ñ‚(:‡ äÒäÔŸ»®~ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04b…yÒ‚(:î äÒäÔÉ츀 disk.drbd1716.out‚0$ disk./.usageGB‚259.1868591308594# memory_percent‚88.3913516998291 disk.nvme0c0n1.in‚4.671875 disk.sda2.out‚0 disk.dm-1.out‚ 357.71484375 disk.sda1.in‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 2.206291e+06 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 3.06640625( disk.nvme0c0n1.out‚25885.31396484375 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.sda1.out‚0& disk.nvme0n1.out‚25885.31396484375( proc.memory.availableMB‚ 1901.9609375# proc.memory.rssMB‚ 2772.39453125# disk.dm-0.out‚25885.31396484375 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda.out‚ 357.71484375 disk.rbd0.out‚0* proc.memory.percent‚16.921353340148926 disk.sda3.out‚ 357.71484375 disk.drbd2265.out‚0 disk.drbd2266.in‚0 cpu‚25.35163769537167 network.sent‚ 1.7734252e+07) disk./.usagePercent‚17.39424758853191 disk.sda.in‚ 126.77734375F*’Ó‚(:‡ óÒäÔá»®~ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.437' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.8874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02ñ¦IuÔ‚(:ê óÒäÔªÍÍ* proc.memory.percent‚16.921353340148926# disk.dm-0.out‚25922.18310546875 disk.sda.in‚ 126.77734375 disk.drbd2080.out‚0% disk./.usageGB‚259.20644760131836 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1901.95703125# proc.memory.rssMB‚ 2772.39453125 disk.dm-0.in‚4.671875 proc.cpu.threads‚17 network.sent‚ 1.7831451e+07* disk./.usagePercent‚17.395562187220236 disk.rbd1.out‚ 3.06640625 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 358.28125 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚25922.18310546875 disk.sda2.in‚0 disk.dm-1.out‚ 358.28125 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 network.recv‚ 2.211393e+06& disk.nvme0n1.out‚25922.18310546875 disk.nvme0c0n1.in‚4.671875 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0$ memory_percent‚88.39137554168701 cpu‚25.350920003043164 disk.rbd1.in‚0 disk.sda.out‚ 358.28125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.drbd2265.out‚0p÷ì‹2Õ- _wandbruntime‚6024  _runtime‚6024j¿YÇÖÁ % _timestamp‚1.7884225242853863e+09 _runtime‚6024.022980909 _step‚618 & lr-AdamW/pg1‚6.268423281832797e-05 & lr-AdamW/pg2‚6.268423281832797e-05 trainer/global_step‚7549Ζº®Ì×Æ trainer/global_step‚7549 % _timestamp‚1.7884225242853863e+09 _runtime‚6024.022980909 _step‚618 & lr-AdamW/pg1‚6.268423281832797e-05 & lr-AdamW/pg2‚6.268423281832797e-05êò õ2Ø- _wandbruntime‚6024  _runtime‚6024ÔUüæ'ÙàO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6327381946368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6326725583872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142620082 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6637435458560 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64132612 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6304085597204 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 6024.02394042 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486438163457536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814255616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142600044 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439404106674688 % _timestamp‚1.7884225242861545e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252513108 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36164872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚486438344730112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚619 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109913064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64132614 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488621056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100297939 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚445731488621056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100297939 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439404106674688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252533148 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445730814255616 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109913066 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109913064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 6024.02394042 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚434187012839444 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚427882927242240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚427882910202880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142600044 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚486438163457536 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚445730814255616 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142600044 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6327381946368 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚455 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚453 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439404088671744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439404106674688 % _timestamp‚1.7884225242861545e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109913066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ë†t9¿2Û- _wandbruntime‚6024  _runtime‚6024·vo²ñ&ÜëM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64140780 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36169396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638461739520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594412 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100310618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545011712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500685254144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427937831372800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252545578 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618498 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100310176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439460441710592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884225250321648e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414HIÙñö&ÝðM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439460441710592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚486500685254144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142638562 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚427937831372800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6328201387520 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252545578 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109927080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142618498 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100310176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142618498 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252565644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439460441710592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884225250321648e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚427937814333440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64140780 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439460459713536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36169396 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638461739520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252565644 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109927082 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6304247594412 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100310618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109927080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64140782 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6304902032376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6327545011712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109927082 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844ìí’`2Þ- _wandbruntime‚6024  _runtime‚6024æãÅMVßÐ .loss/FRt3/assignment_loss‚2.97180332839489 1loss/SRqqt2/assignment_loss‚2.086508607542193 1loss/FBt3/assignment_loss‚0.18463890526133278 1loss/FBt4/assignment_loss‚0.20770560919800526 .loss/FRt3/detection_loss‚2.611764108017087 epoch‚6 % _timestamp‚1.7884225250342221e+09 _runtime‚6024.771684021 0loss/FRt2/assignment_loss‚3.2842281106859446 .loss/FBt3/detection_loss‚2.726263231701321 0loss/FRt1/assignment_loss‚3.1423309404741633 1loss/FBt1/assignment_loss‚0.32716832966120407 /loss/SRqqt1/detection_loss‚2.09295936414729 /loss/SRqqt2/detection_loss‚2.09711956655657 1loss/SRqqt3/detection_loss‚2.0430142351736627 1loss/SRqqt4/detection_loss‚1.1436202371673239 trainer/global_step‚7549 _step‚621 /loss/FRt4/detection_loss‚1.8041991658508778 .loss/FBt4/detection_loss‚0.544382305094041 0loss/FRt4/assignment_loss‚1.5539125168323518 0loss/FBt2/assignment_loss‚0.3460222310545346 /loss/FBt2/detection_loss‚2.3724620558998804 %loss/total_loss‚43.18210646582349 /loss/FRt2/detection_loss‚2.8891491647809744 /loss/FBt1/detection_loss‚2.1352005871859463 /loss/FRt1/detection_loss‚2.2712097574364054 1loss/SRqqt1/assignment_loss‚1.636238974073659 2loss/SRqqt3/assignment_loss‚1.5471480131770174 1loss/SRqqt4/assignment_loss‚1.163057120456811°Û[àÕ 0loss/FRt2/assignment_loss‚3.2842281106859446 .loss/FBt3/detection_loss‚2.726263231701321 0loss/FRt1/assignment_loss‚3.1423309404741633 1loss/FBt1/assignment_loss‚0.32716832966120407 /loss/SRqqt1/detection_loss‚2.09295936414729 /loss/SRqqt2/detection_loss‚2.09711956655657 1loss/SRqqt3/detection_loss‚2.0430142351736627 1loss/SRqqt4/detection_loss‚1.1436202371673239 trainer/global_step‚7549 _step‚621 /loss/FRt4/detection_loss‚1.8041991658508778 .loss/FBt4/detection_loss‚0.544382305094041 0loss/FRt4/assignment_loss‚1.5539125168323518 0loss/FBt2/assignment_loss‚0.3460222310545346 /loss/FBt2/detection_loss‚2.3724620558998804 %loss/total_loss‚43.18210646582349 /loss/FRt2/detection_loss‚2.8891491647809744 /loss/FBt1/detection_loss‚2.1352005871859463 /loss/FRt1/detection_loss‚2.2712097574364054 1loss/SRqqt1/assignment_loss‚1.636238974073659 2loss/SRqqt3/assignment_loss‚1.5471480131770174 1loss/SRqqt4/assignment_loss‚1.163057120456811 .loss/FRt3/assignment_loss‚2.97180332839489 1loss/SRqqt2/assignment_loss‚2.086508607542193 1loss/FBt3/assignment_loss‚0.18463890526133278 1loss/FBt4/assignment_loss‚0.20770560919800526 .loss/FRt3/detection_loss‚2.611764108017087 epoch‚6 % _timestamp‚1.7884225250342221e+09 _runtime‚6024.771684021íƒmxtá‚(:… ‚ÓäÔº•¬~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.244 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑžM@â‚(:ò ‚ÓäÔ»‹ž disk.dm-1.out‚ 358.81640625 disk.sda2.in‚0$ memory_percent‚88.39614391326904 proc.cpu.threads‚17 network.sent‚ 1.7837581e+07 disk.rbd1.out‚ 3.06640625 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.921353340148926 disk.rbd1.in‚0 disk.sda.out‚ 358.81640625* disk./.usagePercent‚17.396828144477052 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.sda2.out‚0 network.recv‚ 2.216139e+06$ disk./.usageGB‚259.22531127'V"¶=92969 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1901.17578125# proc.memory.rssMB‚ 2772.39453125 disk.sda3.out‚ 358.81640625 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875 disk.drbd1716.out‚0& disk.nvme0n1.out‚25962.92822265625( disk.nvme0c0n1.out‚25962.92822265625 cpu‚25.350177983431298 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚25962.92822265625 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875 ¯>»’ã‚(:‡ ‘ÓäÔÏŠ~ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.193' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò·wÕzä‚(:ï ‘ÓäÔœ¦•€ disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 network.recv‚ 2.222452e+06% disk./.usageGB‚259.17304611206055 disk.drbd2080.out‚0 disk.sda.out‚ 359.3359375& disk.nvme0n1.out‚25999.90869140625 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 359.3359375* proc.memory.percent‚16.921353340148926 disk.dm-0.in‚4.671875# disk.dm-0.out‚25999.90869140625 disk.dm-1.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 proc.cpu.threads‚17 disk.rbd0.in‚0( disk.nvme0c0n1.out‚25999.90869140625 disk.sda2.in‚0# proc.memory.rssMB‚ 2772.39453125 cpu‚25.349404343284032 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.drbd1716.out‚0 network.sent‚ 1.7945984e+07) disk./.usagePercent‚17.39332058525043 disk.dm-1.out‚ 359.3359375 disk.rbd0.out‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1901.64453125 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375$ memory_percent‚88.39328289031982å‚(:‡  ÓäÔô¯ø} gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ñ¨­sæ‚(:è  ÓäÔõÀ÷€ proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.in‚0 cpu‚25.348659851158747 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚26041.76025390625 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.rbd1.out‚ 3.06640625 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1901.88671875# proc.memory.rssMB‚ 2772.39453125 network.sent‚ 1.7952329e+07 disk.sda2.out‚0$ disk./.usageGB‚259.1962471008301 disk.sda1.out‚0# disk.dm-0.out‚26041.76025390625 disk.dm-1.out‚ 359.78125 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.in‚0# memory_percent‚88.3918046951294* disk./.usagePercent‚17.394877623073675 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26041.76025390625 disk.sda2.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921353340148926 network.recv‚ 2.227181e+06 disk.sda.out‚ 359.78125 disk.sda3.out‚ 359.78125 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375^?)52ç- _wandbruntime‚6063  _runtime‚6063ÀpÇèÁ & lr-AdamW/pg1‚6.150977021837288e-05 & lr-AdamW/pg2‚6.150977021837288e-05 trainer/global_step‚7599 % _timestamp‚1.7884225636657562e+09 _runtime‚6063.403304651 _step‚622éôÌéÆ trainer/global_step‚7599 % _timestamp‚1.7884225636657562e+09 _runtime‚6063.403304651 _step‚622 & lr-AdamW/pg1‚6.150977021837288e-05 & lr-AdamW/pg2‚6.150977021837288e-05îÜËeK2ê- _wandbruntime‚6063  _runtime‚6063iÐQç'ëáO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711389696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _runtime‚6063.404281204 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436972383229736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚7599 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430628116730880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36393453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563783449600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100934467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36392991 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563595885568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448590126369280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143526096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143526096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680494627328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6344266498856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680331003392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚623 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100934003 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884225636664743e+09©ü›ì'ìæO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚482883288822272 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64541014 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚436972383229736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6368367752192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367711389696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442221740614144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6368367752192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚448589452003840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254139960 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 trainer/global_step‚7599 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430628116730880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6344920936612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36393453 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚448589452003840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489563783449600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110613864 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100934467 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36392991 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64541012 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489563595885568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚448590126369280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442221740614144 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254160000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143526096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143526096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442221758617088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6680494627328 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110613866 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482883264882176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436973054706852 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6344266498856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6680331003392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110613866 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚448590126369280 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚623 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100934003 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 % _timestamp‚1.7884225636664743e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110613864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367711389696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _runtime‚6063.404281204 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ï9/.2í- _wandbruntime‚6064  _runtime‚6064’ºo“[îëM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6681198599168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6369187193344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110627880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110627882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚430683020861440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442278093652992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36397965 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143564614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100947147 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100946686 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448647298849280 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626114219008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437028103795260 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6345082933820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530817536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298849280ì7ç'ö&ïðM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚624 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254172430 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚489626114219008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚437028103795260 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254172430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6345082933820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254192496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368530817536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64549180 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚448647298849280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254192496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143564614 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚442278111655936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6681198599168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110627882 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6369187193344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚482944939559936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚448646624470528 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚437028775272584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110627880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚482944915619840 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442278093652992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6369187193344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110627880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110627882 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430683037900800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚430683020861440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442278093652992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143544550 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36397965 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143564614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100947147 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100946686 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚448647298849280 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111314666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697195520 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451448764117504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64949412 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472186 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101570997 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6723268622336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433373323258880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485965825705472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889Géóµ4üåO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439758425097144 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6385756276020 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6409353558016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚7649 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚627 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445039392556544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144452148 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492689070387712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445039410559488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451448764117504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64949412 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111314666 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144472186 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 101570997 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚64949414 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6723268622336 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433373323258880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451448089752064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚485965825705472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144472186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101570551 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6385101838264 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6408697195520 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚439759096574260 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6409353558016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111314664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚6102.455708754 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111314666 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6408697195520 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111328682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101583240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144470602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439814817139992 trainer/global_step‚7649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144470602 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262218752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111328680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439814145662668 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884226034505887e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172999168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101583669 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36625660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751593975296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724296694272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6386572711192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451505936597504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486027452503040Ÿ‹Pö&ÿðM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101583240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144470602 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439814817139992 trainer/global_step‚7649 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144470602 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451505262218752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111328680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445095763598336 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439814145662668 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884226034505887e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6410172999168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101583669 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433428227389440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36625660 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚492751593975296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6724296694272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6386572711192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451505936597504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255819348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚486027452503040 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144490666 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445095745595392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6409516623360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚492751773137408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6409516623360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255819348 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451505936597504 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144490666 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36626087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚445095763598336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚628 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111328682 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255799282 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111328680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433428244428800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6410172999168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414ôB{A2€- _wandbruntime‚6103  _runtime‚6103jÁ½ÄXÒ epoch‚6 trainer/global_step‚7649 % _timestamp‚1.7884226034521062e+09 .loss/FRt1/detection_loss‚2.396286534635644 .loss/FBt1/detection_loss‚2.356770540277163 _step‚629 /loss/FRt3/assignment_loss‚2.983082826795249 2loss/SRqqt3/assignment_loss‚1.5926162396488641 /loss/FBt1/assignment_loss‚0.317220179397206 1loss/FBt2/assignment_loss‚0.40395844308659434 0loss/SRqqt3/detection_loss‚1.744907579212277 _runtime‚6103.189530405 0loss/FRt1/assignment_loss‚3.2990942879727014 /loss/FRt4/assignment_loss‚1.240320572038976 2loss/SRqqt4/assignment_loss‚1.1171199959030167 1loss/FBt3/assignment_loss‚0.43326101880520584 .loss/FRt3/detection_loss‚2.067369028540521 /loss/FRt4/detection_loss‚2.3077747361598218 .loss/FBt2/detection_loss‚3.769212320446968 %loss/total_loss‚43.90097621855841 2loss/SRqqt2/assignment_loss‚2.0047744632218825 1loss/FBt4/assignment_loss‚0.19404250579774293 /loss/FRt2/detection_loss‚2.9409022408147014 2loss/SRqqt1/assignment_loss‚1.3845323298678354 1loss/SRqqt2/detection_loss‚2.2766437259587375 0loss/SRqqt4/detection_loss‚0.890221548318449 /loss/FBt3/detection_loss‚2.6275137424468995 /loss/FRt2/assignment_loss‚2.961421937711777 1loss/SRqqt1/detection_loss‚1.8405500940150685 /loss/FBt4/detection_loss‚0.7513793274851107Ÿªn]‚× epoch‚6 trainer/global_step‚7649 % _timestamp‚1.7884226034521062e+09 .loss/FRt1/detection_loss‚2.396286534635644 .loss/FBt1/detection_loss‚2.356770540277163 _step‚629 /loss/FRt3/assignment_loss‚2.983082826795249 2loss/SRqqt3/assignment_loss‚1.5926162396488641 /loss/FBt1/assignment_loss‚0.317220179397206 1loss/FBt2/assignment_loss‚0.40395844308659434 0loss/SRqqt3/detection_loss‚1.744907579212277 _runtime‚6103.189530405 0loss/FRt1/assignment_loss‚3.2990942879727014 /loss/FRt4/assignment_loss‚1.240320572038976 2loss/SRqqt4/assignment_loss‚1.1171199959030167 1loss/FBt3/assignment_loss‚0.43326101880520584 .loss/FRt3/detection_loss‚2.067369028540521 /loss/FRt4/detection_loss‚2.3077747361598218 .loss/FBt2/detection_loss‚3.769212320446968 %loss/total_loss‚43.90097621855841 2loss/SRqqt2/assignment_loss‚2.0047744632218825 1loss/FBt4/assignment_loss‚0.19404250579774293 /loss/FRt2/detection_loss‚2.9409022408147014 2loss/SRqqt1/assignment_loss‚1.3845323298678354 1loss/SRqqt2/detection_loss‚2.2766437259587375 0loss/SRqqt4/detection_loss‚0.890221548318449 /loss/FBt3/detection_loss‚2.6275137424468995 /loss/FRt2/assignment_loss‚2.961421937711777 1loss/SRqqt1/detection_loss‚1.8405500940150685 /loss/FBt4/detection_loss‚0.7513793274851107õ±¨Kƒ‚(:… ÍÓäÔù߇~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.36' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v™c|„‚(:ñ ÍÓäÔ­€* disk./.usagePercent‚17.393652115007267 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 361.25390625& disk.nvme0n1.out‚26157.69873046875# disk.dm-0.out‚26157.69873046875$ memory_percent‚90.80719947814941 network.recv‚ 2.240073e+06 disk.sda2.out‚0( proc.memory.availableMB‚ 1506.1484375* proc.memory.percent‚16.426801681518555 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 cpu‚25.346473790272217 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚26157.69873046875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 361.25390625 disk.rbd1.out‚ 3.06640625 network.sent‚ 1.8071123e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2691.3671875% disk./.usageGB‚259.17798614501953 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda3.out‚ 361.25390625 proc.cpu.threads‚17Šè’…‚(:‡ ÜÓäÔÈ¡º~ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“®éÏq†‚(:æ ÜÓäÔ¼ä disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 361.7578125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 disk.sda.out‚ 361.7578125" proc.memory.rssMB‚ 2691.3671875 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875$ disk.nvme0n1.out‚26197.537109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚26197.537109375 network.recv‚ 2.246665e+06 disk.rbd0.in‚0 disk.sda3.out‚ 361.7578125* proc.memory.percent‚16.426801681518555 network.sent‚ 1.818016e+07$ disk./.usageGB‚259.2137756347656! disk.dm-0.out‚26197.537109375 disk.drbd2266.out‚0$ memory_percent‚90.80736637115479 disk.nvme0c0n1.in‚4.671875) disk./.usagePercent‚17.39605397769351 disk.rbd0.out‚0 disk.rbd1.out‚ 3.06640625) proc.memory.availableMB‚ 1506.12109375 cpu‚25.345736949770295 proc.cpu.threads‚17A•Þ‡‚(:… ëÓäÔ¸Õ¥~ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.302 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ö­Oˆ‚(:å ëÓäÔŠ³µ disk.sda3.out‚ 362.40234375 disk.drbd2080.out‚0 disk.sda.out‚ 362.40234375 disk.nvme0n1.in‚4.671875" proc.memory.rssMB‚ 2691.3671875 cpu‚25.34502007972777 disk.sda1.in‚0$ disk.nvme0n1.out‚26234.224609375 network.recv‚ 2.250679e+06* disk./.usagePercent‚17.39764634455ݾÊG!6485$ disk./.usageGB‚259.2375030517578 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 362.40234375 disk.rbd0.out‚0$ memory_percent‚90.80734252929688 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.671875& disk.nvme0c0n1.out‚26234.224609375$ proc.memory.availableMB‚1506.125 proc.cpu.threads‚17 network.sent‚ 1.8185868e+07 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875 disk.sda.in‚ 126.77734375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625 disk.drbd1716.in‚0* proc.memory.percent‚16.426801681518555 disk.sda3.in‚ 126.77734375! disk.dm-0.out‚26234.224609375 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0Pï÷=2‰- _wandbruntime‚6141  _runtime‚6141No–.ÉŠÃ trainer/global_step‚7699 % _timestamp‚1.7884226421022403e+09 _runtime‚6141.839797858 _step‚630 ' lr-AdamW/pg1‚5.9175364205317335e-05 ' lr-AdamW/pg2‚5.9175364205317335e-05WL8Î‹È ' lr-AdamW/pg1‚5.9175364205317335e-05 ' lr-AdamW/pg2‚5.9175364205317335e-05 trainer/global_step‚7699 % _timestamp‚1.7884226421022403e+09 _runtime‚6141.839797858 _step‚630öÌv\â2Œ- _wandbruntime‚6141  _runtime‚6141Ä–@ç'áO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚442545138441668 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚436118546826240 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257393664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6766359845888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6425937177672 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145378200 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447857062501888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚495814698494464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112015466 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102208300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36850488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36850940 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6141.840814136 _step‚631 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814879767040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚7699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454307401865728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447857044498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6766517178368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884226421030262e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398238 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489048362588672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6450339363840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112015464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Jõì'ŽæO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36850940 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65357814 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6141.840814136 _step‚631 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚495814879767040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489048338648576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 trainer/global_step‚7699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚454307401865728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447857044498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6766517178368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884226421030262e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145398238 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65357812 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489048362588672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6450339363840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112015464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚454306727500288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36850488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0÷Èež2- _wandbruntime‚6142  _runtime‚6142[tÎ ñ&ëM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6451158804992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767380054528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426134 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447913397537792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6142.581952697 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426134 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36855513 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877393380864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454364574345728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36855054 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6427408050600 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6767210152448 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899966976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102221034 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚632 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416718%÷u¨‘ðM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6451158804992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚489109989386240 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6767210152448 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257446200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStats¶îN#Monitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚454364574345728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚454363899966976 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102221034 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚632 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145416718 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6450502429184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6451158804992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6767380054528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65365980 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257426134 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚447913397537792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚442600859007400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚7699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6142.581952697 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257446200 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112029480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257426134 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36855513 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145396654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447913415540736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145396654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚436173433917440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚447913415540736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112029482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65365982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447913397537792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚495877393380864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112029482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚454364574345728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36855054 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145416718 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6427408050600 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ø75–x2’- _wandbruntime‚6142  _runtime‚61422¼òZ“Ô 0loss/FBt3/assignment_loss‚0.1276032531181541 1loss/SRqqt3/detection_loss‚1.4879747097276979 1loss/SRqqt4/detection_loss‚0.8633611033049723 .loss/FBt3/detection_loss‚2.934106268666007 &loss/total_loss‚41.361999940890044 _step‚633 /loss/FRt3/assignment_loss‚2.424374336004257 0loss/SRqqt1/detection_loss‚1.995616885168212 0loss/FRt4/assignment_loss‚1.2656474100782515 1loss/SRqqt4/assignment_loss‚1.043081454194584 epoch‚6 /loss/FRt1/detection_loss‚2.1132507873516455 /loss/FRt2/detection_loss‚2.5524932756656553 /loss/FRt3/detection_loss‚2.4916897839307786 /loss/FBt1/detection_loss‚2.4552702582799473 % _timestamp‚1.7884226428459077e+09 .loss/FRt2/assignment_loss‚3.46088760073592 2loss/SRqqt1/assignment_loss‚1.9047893503946918 1loss/FBt2/assignment_loss‚0.32601122320629655 /loss/FRt4/detection_loss‚1.8230348297573151 /loss/FBt4/detection_loss‚0.5471990905520131 /loss/FRt1/assignment_loss‚3.250442405541738 1loss/FBt4/assignment_loss‚0.07404831845400738 trainer/global_step‚7699 2loss/SRqqt2/assignment_loss‚1.8111917087379352 0loss/SRqqt2/detection_loss‚2.076533943414688 .loss/FBt2/detection_loss‚2.518498682975769 _runtime‚6142.583379059 2loss/SRqqt3/assignment_loss‚1.3726678475698766 1loss/FBt1/assignment_loss‚0.44222541405962634Ãî²_”Ù 0loss/FRt4/assignment_loss‚1.2656474100782515 1loss/SRqqt4/assignment_loss‚1.043081454194584 epoch‚6 /loss/FRt1/detection_loss‚2.1132507873516455 /loss/FRt2/detection_loss‚2.5524932756656553 /loss/FRt3/detection_loss‚2.4916897839307786 /loss/FBt1/detection_loss‚2.4552702582799473 % _timestamp‚1.7884226428459077e+09 .loss/FRt2/assignment_loss‚3.46088760073592 2loss/SRqqt1/assignment_loss‚1.9047893503946918 1loss/FBt2/assignment_loss‚0.32601122320629655 /loss/FRt4/detection_loss‚1.8230348297573151 /loss/FBt4/detection_loss‚0.5471990905520131 /loss/FRt1/assignment_loss‚3.250442405541738 1loss/FBt4/assignment_loss‚0.07404831845400738 trainer/global_step‚7699 2loss/SRqqt2/assignment_loss‚1.8111917087379352 0loss/SRqqt2/detection_loss‚2.076533943414688 .loss/FBt2/detection_loss‚2.518498682975769 _runtime‚6142.583379059 2loss/SRqqt3/assignment_loss‚1.3726678475698766 1loss/FBt1/assignment_loss‚0.44222541405962634 0loss/FBt3/assignment_loss‚0.1276032531181541 1loss/SRqqt3/detection_loss‚1.4879747097276979 1loss/SRqqt4/detection_loss‚0.8633611033049723 .loss/FBt3/detection_loss‚2.934106268666007 &loss/total_loss‚41.361999940890044 _step‚633 /loss/FRt3/assignment_loss‚2.424374336004257 0loss/SRqqt1/detection_loss‚1.995616885168212ùÈše”’•‚(:‡ úÓäÔ›êœ~ gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚132.727' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó˜à§t–‚(:é úÓäÔîЈ disk.drbd2266.out‚0 disk.rbd1.out‚ 3.06640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 362.86328125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2691.3828125* proc.memory.percent‚16.426897048950195 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚4.671875 cpu‚25.344347052145316 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0 disk.dm-1.out‚ 362.86328125 network.sent‚ 1.8195148e+07 disk.nvme0n1.in‚4.671875# disk.nvme0n1.out‚26274.16796875 disk.dm-1.in‚ 126.77734375 network.recv‚ 2.261653e+06 disk.sda.out‚ 362.86328125% disk.nvme0c0n1.out‚26274.16796875 disk.sda3.in‚ 126.77734375 disk.dm-0.out‚26274.16796875$ memory_percent‚90.80784320831299% disk./.usageGB‚259.25121688842773 disk.rbd0.in‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1506.04296875* disk./.usagePercent‚17.398566691642078 disk.rbd0.out‚0 disk.drbd2080.out‚0ˆs[›’—‚(:‡ ‰ÔäÔª²£~ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€P…g˜‚(:Ü ‰ÔäÔéÊ  cpu‚25.343632283981275 network.sent‚ 1.830369e+07 disk.sda1.out‚0 disk.sda2.out‚0# memory_percent‚90.8078670501709 disk.rbd1.out‚ 3.06640625( proc.memory.availableMB‚ 1506.0390625 disk.dm-0.out‚ 26309.796875 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0* proc.memory.percent‚16.426897048950195$ disk./.usageGB‚259.2733612060547 disk.nvme0n1.in‚4.671875 disk.sda3.out‚ 363.5390625 disk.rbd1.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2691.3828125 network.recv‚ 2.267747e+06* disk./.usagePercent‚17.400052815378345 disk.sda.out‚ 363.5390625 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚4.671875 disk.sda1.in‚0! disk.nvme0n1.out‚ 26309.796875 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 363.5390625 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0# disk.nvme0c0n1.out‚ 26309.796875 disk.drbd2080.out‚0ÛÒ’™‚(:‡ ˜ÔäÔÖŒ~ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙŽ1åfš‚(:Û ˜ÔäÔð € disk.dm-1.out‚363.875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.06640625% disk./.usageGB‚259.28801345825195 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd1716.out‚0& disk.nvme0c0n1.out‚26353.705078125 disk.sda1.in‚0) proc.memory.availableMB‚ 1506.63671875 network.recv‚ 2.272296e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda.out‚363.875 cpu‚25.342890720672298 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚90.80421924591064* proc.memory.percent‚16.438007354736328 network.sent‚ 1.8309988e+07 disk.sda3.out‚363.875 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ disk.nvme0n1.out‚26353.705078125 disk.sda.in‚ 126.77734375! proc.memory.rssMB‚ 2693.203125 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.671875! disk.dm-0.out‚26353.705078125 disk.dm-1.in‚ 126.77734375) disk./.usagePercent‚17.40103614031736["Wé2›- _wandbruntime‚6180  _runtime‚6180UØÄÇœÁ & lr-AdamW/pg1‚5.801590727893111e-05 & lr-AdamW/pg2‚5.801590727893111e-05 trainer/global_step‚7749 % _timestamp‚1.7884226809075046e+09 _runtime‚6180.645049539 _step‚634Ô²ÇÞÌÆ _step‚634 & lr-AdamW/pg1‚5.801590727893111e-05 & lr-AdamW/pg2‚5.801590727893111e-05 trainer/global_step‚7749 % _timestamp‚1.7884226809075046e+09 _runtime‚6180.645049539úÇ»ü62ž- _wandbruntime‚6180  _runtime‚6180ì×ç'ŸáO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6809445577728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438863736314880 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445330508831960 trainer/global_step‚7749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102845483 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457165365248512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112716266 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6180.646069905 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039613952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112716264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37079269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6467426954836 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450674714444288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492130875531776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146304252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457166039613952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37078815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 J‚ æO _runtime‚6180.646069905 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146324290 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457166039613952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112716264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37079269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚492130899471872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6467426954836 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚450674714444288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146304252 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457165365248512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚492130875531776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6466772517080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146304252 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457166039613952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37078815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102845483 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457165365248512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112716266 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490668807168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0û4~ í2¡- _wandbruntime‚6181  _runtime‚6181Ê€V3ð&¢êM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112730282 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952044 _runtime‚6181.610563169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212093952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102857753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6468243390008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457223212093952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144610816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788422681872521e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450731049480192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450731049480192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002846130688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810295921152 trainer/global_step‚7749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37083373 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102858217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844c¦‰õ&£ïM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146322706 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6810119727616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146322706 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146342770 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65774382 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457222537715200 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112730282 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6467588952044 _runtime‚6181.610563169 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚492192550209536 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _step‚636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457223212093952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259052986 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491488235008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259073052 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102857753 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6468243390008 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65774380 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450731067483136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438918640445440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457223212093952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492144610816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112730280 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788422681872521e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450731049480192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450731049480192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚499002846130688 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112730280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6810295921152 trainer/global_step‚7749 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112730282 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491488235008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37083373 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 259052986 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445386900874808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102858217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654612992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚639 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6531654612992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 trainer/global_step‚7799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6219.797272569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677362176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ÖÈXì'°æO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66174614 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚495213412414976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6531654612992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚448116550699368 trainer/global_step‚7799 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6219.797272569 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66174612 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚460024677362176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147230304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6532310975488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260647368 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113417066 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6852518461440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37306769 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147250342 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147250342 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103481827 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚460024002996736 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚453492348383744 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6507607856488 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚453492366386688 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460024002996736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6531654612992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚639 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ÿZ˜g2±- _wandbruntime‚6220  _runtime‚6220æù*°ð&²êM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130416640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66182780 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128467811840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37311380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6220.527145853 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502128286552576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6533130416640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103494160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37311819 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448172271264892 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6853223399936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495275063152640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103494601 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268822 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788422720788926e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460081175463424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853380719104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752ëühþ³ïM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6532474040832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚453548719425536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37311380 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6220.527145853 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚502128286552576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚453548719425536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147268822 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6533130416640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260679838 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103494160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441663846973440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚495275087092736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37311819 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚448172271264892 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚453548701422592 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚453548701422592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66182782 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6853223399936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260699904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260699904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚495275063152640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113431082 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147248758 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113431082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103494601 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113431080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147268822 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788422720788926e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460081175463424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853380719104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460081849842176 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6509078729416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6533130416640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66182780 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260679838 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502128467811840€Ã3¸2´- _wandbruntime‚6220  _runtime‚6220×OÐÔPµÊ /loss/FBt3/assignment_loss‚0.392587768069158 /loss/FRt1/detection_loss‚2.1498124768149176 _runtime‚6220.528163324 0loss/SRqqt2/detection_loss‚2.361570200612468 %loss/total_loss‚44.45767684230385 epoch‚6 0loss/FRt3/assignment_loss‚2.7718331130539497 1loss/SRqqt2/assignment_loss‚1.689577405491183 trainer/global_step‚7799 /loss/FRt2/assignment_loss‚4.032017634465144 0loss/FBt2/assignment_loss‚0.5837889450602234 0loss/SRqqt1/detection_loss‚1.903548232637919 /loss/FRt1/assignment_loss‚3.717090324410852 .loss/FRt4/detection_loss‚1.838027204028689 1loss/SRqqt3/detection_loss‚1.6212885103442451 1loss/SRqqt4/detection_loss‚1.1245297731186907 .loss/FBt1/detection_loss‚2.417793869972229 1loss/SRqqt3/assignment_loss‚1.438424847681414 0loss/FBt1/assignment_loss‚0.5011666193604469 0loss/FBt4/assignment_loss‚0.1665088908322071 ,loss/FRt3/detection_loss‚2.5030469749032 /loss/FBt2/detection_loss‚2.5275168895721434 _step‚641 2loss/SRqqt1/assignment_loss‚1.2081265471493587 /loss/FBt3/detection_loss‚3.6788232922554016 1loss/SRqqt4/assignment_loss‚1.015454220227324 /loss/FRt2/detection_loss‚3.1121814113396864 /loss/FBt4/detection_loss‚0.6573108568286093 $ _timestamp‚1.788422720790727e+09 /loss/FRt4/assignment_loss‚1.045650834074387» 8U¶Ï trainer/global_step‚7799 /loss/FRt2/assignment_loss‚4.032017634465144 0loss/FBt2/assignment_loss‚0.5837889450602234 0loss/SRqqt1/detection_loss‚1.903548232637919 /loss/FRt1/assignment_loss‚3.717090324410852 .loss/FRt4/detection_loss‚1.838027204028689 1loss/SRqqt3/detection_loss‚1.6212885103442451 1loss/SRqqt4/detection_loss‚1.1245297731186907 .loss/FBt1/detection_loss‚2.417793869972229 1loss/SRqqt3/assignment_loss‚1.438424847681414 0loss/FBt1/assignment_loss‚0.5011666193604469 0loss/FBt4/assignment_loss‚0.1665088908322071 ,loss/FRt3/detection_loss‚2.5030469749032 /loss/FBt2/detection_loss‚2.5275168895721434 _step‚641 2loss/SRqqt1/assignment_loss‚1.2081265471493587 /loss/FBt3/detection_loss‚3.6788232922554016 1loss/SRqqt4/assignment_loss‚1.015454220227324 /loss/FRt2/detection_loss‚3.1121814113396864 /loss/FBt4/detection_loss‚0.6573108568286093 $ _timestamp‚1.788422720790727e+09 /loss/FRt4/assignment_loss‚1.045650834074387 /loss/FBt3/assignment_loss‚0.392587768069158 /loss/FRt1/detection_loss‚2.1498124768149176 _runtime‚6220.528163324 0loss/SRqqt2/detection_loss‚2.361570200612468 %loss/total_loss‚44.45767684230385 epoch‚6 0loss/FRt3/assignment_loss‚2.7718331130539497 1loss/SRqqt2/assignment_loss‚1.689577405491183XÍL·‚(:’ ÅÔäÔÒ´ƒ~ gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.266' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.653200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Ñ.p¸‚(:å ÅÔäÔ¶‘»€ cpu‚25.340820538389913$ disk./.usageGB‚259.3164978027344 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1506.5390625 network.recv‚ 2.286039e+06 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.sda2.out‚0# disk.dm-0.out‚26465.67138671875 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.402947748575535( disk.nvme0c0n1.out‚26465.67138671875 disk.sda1.in‚0 network.sent‚ 1.8429788e+07 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 366.09375 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4.671875! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328& disk.nvme0n1.out‚26465.67138671875 disk.sda3.out‚ 366.09375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.21484375 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0# memory_percent‚90.8048152923584 disk.sda.out‚ 366.09375¨5‘œ’¹‚(:‡ ÔÔäÔ¤À£~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Zèyº‚(:î ÔÔäÔïϧ€ disk.sda.out‚ 366.67578125$ disk./.usageGB‚259.3205757141113 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 366.67578125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.rbd1.out‚ 3.21484375 network.sent‚ 1.8538756e+07* disk./.usagePercent‚17.403221420629635# disk.dm-0.out‚26505.47998046875 disk.rbd0.in‚0 disk.sda3.out‚ 366.67578125 disk.drbd2080.out‚0$ memory_percent‚90.80629348754883( disk.nvme0c0n1.out‚26505.47998046875* proc.memory.percent‚16.438007354736328 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2693.203125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd2080.in‚0' proc.memory.availableMB‚ 1506.296875 network.recv‚ 2.293117e+06 disk.sda1.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚26505.47998046875 cpu‚25.340125678590205 proc.cpu.threads‚17I>B’»‚(:‡ ãÔäÔÞ¥·~ gpu.0.gpu‚28.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚168.609' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.7218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äþ=m¼‚(:â ãÔäÔ¡ñÏ€ disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0# memory_percent‚90.8048152923584( proc.memory.availableMB‚ 1506.5390625 cpu‚25.33943022818831) disk./.usagePercent‚17.40375775641387 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2693.203125 disk.sda1.out‚0% disk.nvme0n1.out‚26548.5986328125 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875" disk.dm-0.out‚26548.5986328125 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 367.28125 network.recv‚ 2.296698e+06$ disk./.usageGB‚259.3285675048828 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.2470703125 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.438007354736328 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 network.sent‚ 1.8544041e+07 disk.dm-1.out‚ 367.28125 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875' disk.nvme0c0n1.out‚26548.5986328125 disk.sda3.out‚ 367.28125Ï*#2½- _wandbruntime‚6258  _runtime‚6258›¾¯7ǾÁ _runtime‚6258.898613236 _step‚642 & lr-AdamW/pg1‚5.571369368151476e-05 & lr-AdamW/pg2‚5.571369368151476e-05 trainer/global_step‚7849 % _timestamp‚1.7884227591610382e+09/Ì¿Æ & lr-AdamW/pg1‚5.571369368151476e-05 & lr-AdamW/pg2‚5.571369368151476e-05 trainer/global_step‚7849 % _timestamp‚1.7884227591610382e+09 _runtime‚6258.898613236 _step‚642‚ü"» 2À- _wandbruntime‚6258  _runtime‚6258¨/±Ûç'ÁáO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114117866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6573296781312 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104119805 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚498295973238272 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462883315110400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462883315110400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6572640418816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚505191260179968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148176394 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37536791 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6258.899482697 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6572640418816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6548443195896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456310000326144 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6895470311424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6573296781312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚7849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148176394 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚505191443549696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444354149370880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884227591617544e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66583012 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6895310881792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104119346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148176394 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚505191443549696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444354149370880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148156356 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456310000326144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66583014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884227591617544e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462882640744960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66583012 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6895310881792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444354166410240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262294260 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104119346 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚643 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114117864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456310018329088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148176394 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148156356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37536791 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6258.899482697ƒX^Bx2Ã- _wandbruntime‚6259  _runtime‚6259KDÎñ&ÄëM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚644 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148174810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462939813211648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37541315 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6549914068824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104132037 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚6259.632970219 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459846656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306690 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6896152095232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630860 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6573459846656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884227598946342e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚505253752131072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262326756 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456366371367936 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194874 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148174810 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450958313132300 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116222464Ͼæö&ÅðM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6573459846656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262306690 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114131880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6896152095232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6549259630860 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6573459846656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884227598946342e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚505253752131072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66591180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262326756 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456366371367936 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148194874 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148174810 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114131882 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚450958313132300 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6574116222464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚450958984609624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚644 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148174810 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462939813211648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37541315 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456366353364992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 200133632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚444409053501440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6549914068824 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262326756 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104132037 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176193536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚6259.632970219„ÄÇé§2Æ- _wandbruntime‚6259  _runtime‚6259*±WVÇÐ 1loss/FBt2/assignment_loss‚0.28566390519959634 1loss/FBt4/assignment_loss‚0.13015091995268713 /loss/FRt1/assignment_loss‚3.705016199180058 2loss/SRqqt2/assignment_loss‚1.9906859123148024 .loss/FRt2/detection_loss‚3.148360511352276 0loss/SRqqt2/detection_loss‚2.310186524875462 1loss/SRqqt3/detection_loss‚1.6464921049773693 1loss/SRqqt1/assignment_loss‚1.456672345166139 -loss/FBt4/detection_loss‚0.71483679998251 1loss/FBt3/assignment_loss‚0.19447581193526275 .loss/FRt1/detection_loss‚2.500477936438152 1loss/SRqqt4/detection_loss‚1.1221399750432064 /loss/FBt2/detection_loss‚3.0127101710864475 epoch‚6 _runtime‚6259.634522639 _step‚645 0loss/FRt2/assignment_loss‚3.7177799323509477 0loss/FBt1/assignment_loss‚0.3592048707817282 1loss/SRqqt1/detection_loss‚1.9564626074972606 .loss/FBt3/detection_loss‚2.681752989689509 /loss/FRt3/assignment_loss‚2.994082619535162 2loss/SRqqt3/assignment_loss‚1.2547153361998167 .loss/FRt3/detection_loss‚2.341887670628568 .loss/FBt1/detection_loss‚2.216630986758641 trainer/global_step‚7849 0loss/FRt4/assignment_loss‚1.3853934420479668 /loss/FRt4/detection_loss‚1.7835710337178574 2loss/SRqqt4/assignment_loss‚1.0881119236863892 %loss/total_loss‚43.99746253039782 $ _timestamp‚1.788422759897047e+094K!ï[ÈÕ .loss/FBt3/detection_loss‚2.681752989689509 /loss/FRt3/assignment_loss‚2.994082619535162 2loss/SRqqt3/assignment_loss‚1.2547153361998167 .loss/FRt3/detection_loss‚2.341887670628568 .loss/FBt1/detection_loss‚2.216630986758641 trainer/global_step‚7849 0loss/FRt4/assignment_loss‚1.3853934420479668 /loss/FRt4/detection_loss‚1.7835710337178574 2loss/SRqqt4/assignment_loss‚1.0881119236863892 %loss/total_loss‚43.99746253039782 $ _timestamp‚1.788422759897047e+09 1loss/FBt2/assignment_loss‚0.28566390519959634 1loss/FBt4/assignment_loss‚0.13015091995268713 /loss/FRt1/assignment_loss‚3.705016199180058 2loss/SRqqt2/assignment_loss‚1.9906859123148024 .loss/FRt2/detection_loss‚3.148360511352276 0loss/SRqqt2/detection_loss‚2.310186524875462 1loss/SRqqt3/detection_loss‚1.6464921049773693 1loss/SRqqt1/assignment_loss‚1.456672345166139 -loss/FBt4/detection_loss‚0.71483679998251 1loss/FBt3/assignment_loss‚0.19447581193526275 .loss/FRt1/detection_loss‚2.500477936438152 1loss/SRqqt4/detection_loss‚1.1221399750432064 /loss/FBt2/detection_loss‚3.0127101710864475 epoch‚6 _runtime‚6259.634522639 _step‚645 0loss/FRt2/assignment_loss‚3.7177799323509477 0loss/FBt1/assignment_loss‚0.3592048707817282 1loss/SRqqt1/detection_loss‚1.9564626074972606…çã’6’É‚(:‡ òÔäÔýÚÑ~ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T~«XzÊ‚(:ï òÔäÔÇçÓ€ disk.dm-1.out‚ 367.91796875 disk.drbd2265.out‚0 disk.drbd2080.in‚0$ memory_percent‚90.80502986907959) proc.memory.availableMB‚ 1506.50390625$ disk./.usageGB‚259.3446960449219 disk.dm-1.in‚ 126.77734375 network.recv‚ 2.301345e+06 disk.sda1.in‚0 disk.sda2.in‚0 network.sent‚ 1.8550045e+07 disk.drbd1716.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚17.404840156268637 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875% disk.nvme0n1.out‚26583.5830078125 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875' disk.nvme0c0n1.out‚26583.5830078125 disk.sda2.out‚0 disk.sda3.out‚ 367.91796875" disk.dm-0.out‚26583.5830078125 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.rbd1.out‚ 3.2470703125 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.438007354736328 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 367.91796875! proc.memory.rssMB‚ 2693.203125 cpu‚25.338780048918395o¢àHË‚(:„ ÕäÔ•æ˜~ gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vH=§|Ì‚(:ñ ÕäÔ ²·€ disk.drbd1716.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚26622.52783203125 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 network.sent‚ 1.8658466e+07 disk.drbd2266.in‚0 disk.sda.out‚ 368.41015625 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 368.41015625 network.recv‚ 2.307489e+06 disk.rbd1.out‚ 3.31005859375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.drbd2080.out‚0$ memory_percent‚90.80500602722168 disk.sda.in‚ 126.77734375$ disk./.usageGB‚259.3521385192871 disk.dm-0.in‚4.671875# disk.dm-0.out‚26622.52783203125 disk.dm-1.in‚ 126.77734375& disk.nvme0n1.out‚26622.52783203125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2266.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 1506.5078125! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328) disk./.usagePercent‚17.40533962696794 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 368.41015625 disk.drbd2265.in‚0 cpu‚25.338055958601174ùi)2Í- _wandbruntime‚6297  _runtime‚6297ᶠÄξ $ _timestamp‚1.788422798139384e+09 _runtime‚6297.877143525 _step‚646 % lr-AdamW/pg1‚5.45714167883477e-05 % lr-AdamW/pg2‚5.45714167883477e-05 trainer/global_step‚7899ÔrÏÉÏà $ _timestamp‚1.788422798139384e+09 _runtime‚6297.877143525 _step‚646 % lr-AdamW/pg1‚5.45714167883477e-05 % lr-AdamW/pg2‚5.45714167883477e-05 trainer/global_step‚7899†ðÑšÏ2Ð- _wandbruntime‚6297  _runtime‚6297 {!@mÑàO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6938285968384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453689305911300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6589932973060 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7899 _step‚647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278493184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104756712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282587136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626224640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6938285968384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149102446 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149082408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66991412 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453689305911300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6589932973060 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465741278493184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚7899 _step‚647 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465741952858624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465741278493184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104756712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614282587136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6613626224640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263953608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114832680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6614445652480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100862 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6939160310272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104768971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379476391424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37769839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 trainer/global_step‚7899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6939315532288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615102028288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37769391 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263933542 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66999582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114832680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465799125338624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465799125338624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6590094970268 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114832682 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚453745026477032 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263933542 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459184005307392 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149120926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263953608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114832680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚447154277068800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bÌ Öž ytes.small_pool.freed‚ 6614445652480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465798450959872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149100862 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚501440136919040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6939160310272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104768971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508379476391424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453744354999708 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149100862 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37769839 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚459184023310336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 trainer/global_step‚7899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615102028288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚459184023310336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263953608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6939315532288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0ˆ¨Rˆ¶2Ö- _wandbruntime‚6298  _runtime‚6298±®ÂX×Ò 0loss/FRt1/assignment_loss‚3.0335978879815055 /loss/FRt2/assignment_loss‚3.162428555950042 2loss/SRqqt2/assignment_loss‚2.2897831345597903 1loss/SRqqt2/detection_loss‚2.2890982737143832 .loss/FBt2/detection_loss‚2.995496165752411 .loss/FBt3/detection_loss‚2.900753637154897 /loss/FRt1/detection_loss‚2.4174877178101313 epoch‚6 2loss/SRqqt3/assignment_loss‚1.4388364745955915 2loss/SRqqt4/assignment_loss‚0.9864188784458598 1loss/FBt4/assignment_loss‚0.16696102194751958 /loss/FRt3/detection_loss‚2.3919249900616704 % _timestamp‚1.7884227988768666e+09 /loss/FRt2/detection_loss‚2.7990582969880875 .loss/FBt4/detection_loss‚0.672666105037802 trainer/global_step‚7899 0loss/FRt3/assignment_loss‚3.1745778247714043 0loss/FRt4/assignment_loss‚1.7624367678707296 1loss/SRqqt4/detection_loss‚1.1117734334385785 %loss/total_loss‚44.20571371488915 2loss/SRqqt1/assignment_loss‚1.5197202580402027 0loss/FBt1/assignment_loss‚0.2324166268663248 0loss/FBt3/assignment_loss‚0.2432171914539746 1loss/SRqqt1/detection_loss‚2.2238520653666676 -loss/FRt4/detection_loss‚1.94851534106684 _step‚649 1loss/FBt2/assignment_loss‚0.34112482128927013 0loss/SRqqt3/detection_loss‚1.796742644160986 -loss/FBt1/detection_loss‚2.30682560056448 _runtime‚6298.6143271720ª—†]Ø× 2loss/SRqqt2/assignment_loss‚2.2897831345597903 1loss/SRqqt2/detection_loss‚2.2890982737143832 .loss/FBt2/detection_loss‚2.995496165752411 .loss/FBt3/detection_loss‚2.900753637154897 /loss/FRt1/detection_loss‚2.4174877178101313 epoch‚6 2loss/SRqqt3/assignment_loss‚1.4388364745955915 2loss/SRqqt4/assignment_loss‚0.9864188784458598 1loss/FBt4/assignment_loss‚0.16696102194751958 /loss/FRt3/detection_loss‚2.3919249900616704 % _timestamp‚1.7884227988768666e+09 /loss/FRt2/detection_loss‚2.7990582969880875 .loss/FBt4/detection_loss‚0.672666105037802 trainer/global_step‚7899 0loss/FRt3/assignment_loss‚3.1745778247714043 0loss/FRt4/assignment_loss‚1.7624367678707296 1loss/SRqqt4/detection_loss‚1.1117734334385785 %loss/total_loss‚44.20571371488915 2loss/SRqqt1/assignment_loss‚1.5197202580402027 0loss/FBt1/assignment_loss‚0.2324166268663248 0loss/FBt3/assignment_loss‚0.2432171914539746 1loss/SRqqt1/detection_loss‚2.2238520653666676 -loss/FRt4/detection_loss‚1.94851534106684 _step‚649 1loss/FBt2/assignment_loss‚0.34112482128927013 0loss/SRqqt3/detection_loss‚1.796742644160986 -loss/FBt1/detection_loss‚2.30682560056448 _runtime‚6298.614327172 0loss/FRt1/assignment_loss‚3.0335978879815055 /loss/FRt2/assignment_loss‚3.162428555950042‰'IàJ’Ù‚(:‡ ÕäÔ”™~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚172.731' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.5462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0klqÚ‚(:æ ÕäÔÂ’¸€ network.recv‚ 2.312396e+06 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.438007354736328 network.sent‚ 1.8664909e+07 disk.dm-0.in‚4.671875" disk.dm-0.out‚26659.4248046875 disk.drbd2266.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚17.405388268399445$ disk./.usageGB‚259.3528633117676 disk.rbd0.in‚0' disk.nvme0c0n1.out‚26659.4248046875 disk.drbd2265.out‚0 disk.dm-1.out‚ 368.90625 disk.drbd1716.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚26659.4248046875 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 368.90625 disk.nvme0c0n1.in‚4.671875 disk.sda3.out‚ 368.90625 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.31005859375$ memory_percent‚90.80538749694824 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 1506.4453125! proc.memory.rssMB‚ 2693.203125 cpu‚25.337372917806942 disk.nvme0n1.in‚4.671875 disk.sda1.in‚0£IÍ’Û‚(:‡ ŸÕäÔ‚Ç°~ gpu.0.gpu‚50.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×1µIyÜ‚(:î ŸÕäÔží¦€ disk.drbd2265.out‚0* proc.memory.percent‚16.438007354736328 proc.cpu.threads‚17# disk.dm-0.out‚26706.40283203125 disk.dm-1.out‚ 369.3515625 disk.drbd1716.out‚0$ memory_percent‚90.80541133880615 network.recv‚ 2.317708e+06* disk./.usagePercent‚17.405869050548752 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚26706.40283203125 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 369.3515625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 369.3515625 network.sent‚ 1.8772868e+07 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875! proc.memory.rssMB‚ 2693.203125 cpu‚25.336733673931505 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚26706.40283203125) proc.memory.availableMB‚ 1506.44140625$ disk./.usageGB‚259.3600273132324 disk.sda2.out‚0 disk.dm-0.in‚4.671875 disk.rbd0.in‚0î²Ý‚(:’ ®ÕäÔÑž†~ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.729' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.345800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š}¹yÞ‚(:î ®ÕäÔŸ°¡€ disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0 disk.sda3.out‚ 370.17578125* proc.memory.percent‚16.438007354736328 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875 cpu‚25.336018030802997 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚ 2.321938e+06# disk.dm-0.out‚26741.34033203125 disk.drbd2266.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚26741.34033203125! proc.memory.rssMB‚ 2693.203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875 disk.sda3.in‚ 126.77734375 network.sent‚ 1.8778952e+07* disk./.usagePercent‚17.406619408631606% disk./.usageGB‚259.37120819091797 disk.sda.out‚ 370.17578125 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0$ memory_percent‚90.80696105957031% proc.memory.availableMB‚ 1506.1875 disk.rbd1.in‚0 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚26741.34033203125 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-1.out‚ 370.17578125 disk.sda1.in‚0 disk.drbd1716.out‚0³˜DK2ß- _wandbruntime‚6336  _runtime‚6336¢CMÝÇàÁ & lr-AdamW/pg1‚5.343534431172533e-05 & lr-AdamW/pg2‚5.343534431172533e-05 trainer/global_step‚7949 % _timestamp‚1.7884228370195584e+09 _runtime‚6336.757148777 _step‚650j°á ÌáÆ trainer/global_step‚7949 % _timestamp‚1.7884228370195584e+09 _runtime‚6336.757148777 _step‚650 & lr-AdamW/pg1‚5.343534431172533e-05 & lr-AdamW/pg2‚5.343534431172533e-05Š÷¹G2â- _wandbruntime‚6336  _runtime‚6336û`ëç'ãáO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚511442609009664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚511442423542784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268392960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37995582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37996034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028498 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6336.758122756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590606848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461945304210944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612030464 trainer/global_step‚7949 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449844562426880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 _step‚651 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6981562004992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115519466 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6655268392960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115519464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37995582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468599916241408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6655268392960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37996034 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150008460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6654612030464 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150028498 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6336.758122756 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449844579466240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468600590606848 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚461945304210944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6654612030464 trainer/global_step‚7949 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449844562426880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67399814 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115519466 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 265527924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265547964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150008460 _step‚651 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6630768312468 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115519464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504461023064576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚468599916241408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504461047004672 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6981562004992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456474676301592 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115519466 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150028498 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511504971924480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462001675252736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026914 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67407982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884228377320662e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 164482048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6337.470250729 _step‚652 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105408169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6982298122240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6982438664192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 140541952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511505136406528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431458304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736èp|ö&çðM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265580460 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115533480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 164482048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115533482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6337.470250729 _step‚652 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150046978 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150046978 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504522697742336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265560394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6655431458304 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105408169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6982298122240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449899483596800 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6656087834112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6982438664192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 140541952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456530396867116 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6630930309676 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265560394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511505136406528 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6655431458304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462001675252736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38000619 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚468657763086848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115533482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504522673802240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265580460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚511504971924480 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462001675252736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150026914 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67407982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67407980 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884228377320662e+09Œ^0ÿ12è- _wandbruntime‚6337  _runtime‚6337ÂýÅZéÔ 0loss/FRt4/assignment_loss‚1.7914043957111883 /loss/FBt1/detection_loss‚2.9170703440904617 epoch‚6 0loss/FBt4/assignment_loss‚0.1359450640207521 .loss/FRt4/detection_loss‚1.653990794338432 /loss/FBt4/detection_loss‚0.6043454228201881 % _timestamp‚1.7884228377338362e+09 _runtime‚6337.471279782 /loss/FBt3/detection_loss‚2.0283185094594955 %loss/total_loss‚43.33429157912372 trainer/global_step‚7949 /loss/FRt2/detection_loss‚2.9801923676773354 1loss/SRqqt1/detection_loss‚1.7604920598593625 _step‚653 2loss/SRqqt1/assignment_loss‚1.6876235665922814 2loss/SRqqt3/assignment_loss‚1.4505219862011134 .loss/FRt1/detection_loss‚2.267178338766098 .loss/FRt3/detection_loss‚2.802578851099937 1loss/SRqqt2/detection_loss‚2.3195479388038316 1loss/FBt3/assignment_loss‚0.04602859041187912 1loss/SRqqt3/detection_loss‚1.6839709092004627 0loss/FRt2/assignment_loss‚3.5161490926036127 /loss/FRt3/assignment_loss‚3.403002925457493 1loss/SRqqt4/assignment_loss‚1.092557003782665 1loss/FBt1/assignment_loss‚0.22905178473590126 0loss/FBt2/assignment_loss‚0.0338502157246694 1loss/SRqqt4/detection_loss‚0.8059278774212618 /loss/FRt1/assignment_loss‚2.912812719080183 2loss/SRqqt2/assignment_loss‚1.9934657199515238 .loss/FBt2/detection_loss‚3.218265101313591PÄG_êÙ 0loss/FRt2/assignment_loss‚3.5161490926036127 /loss/FRt3/assignment_loss‚3.403002925457493 1loss/SRqqt4/assignment_loss‚1.092557003782665 1loss/FBt1/assignment_loss‚0.22905178473590126 0loss/FBt2/assignment_loss‚0.0338502157246694 1loss/SRqqt4/detection_loss‚0.8059278774212618 /loss/FRt1/assignment_loss‚2.912812719080183 2loss/SRqqt2/assignment_loss‚1.9934657199515238 .loss/FBt2/detection_loss‚3.218265101313591 0loss/FRt4/assignment_loss‚1.7914043957111883 /loss/FBt1/detection_loss‚2.9170703440904617 epoch‚6 0loss/FBt4/assignment_loss‚0.1359450640207521 .loss/FRt4/detection_loss‚1.653990794338432 /loss/FBt4/detection_loss‚0.6043454228201881 % _timestamp‚1.7884228377338362e+09 _runtime‚6337.471279782 /loss/FBt3/detection_loss‚2.0283185094594955 %loss/total_loss‚43.33429157912372 trainer/global_step‚7949 /loss/FRt2/detection_loss‚2.9801923676773354 1loss/SRqqt1/detection_loss‚1.7604920598593625 _step‚653 2loss/SRqqt1/assignment_loss‚1.6876235665922814 2loss/SRqqt3/assignment_loss‚1.4505219862011134 .loss/FRt1/detection_loss‚2.267178338766098 .loss/FRt3/detection_loss‚2.802578851099937 1loss/SRqqt2/detection_loss‚2.3195479388038316 1loss/FBt3/assignment_loss‚0.04602859041187912 1loss/SRqqt3/detection_loss‚1.68397090920046271h$ç’ë‚(:‡ ½ÕäÔÞã~ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¬Frì‚(:ç ½ÕäÔŸ»‘ network.recv‚ 2.325772e+06$ disk./.usageGB‚259.3950538635254 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚26771.0888671875 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0" disk.dm-0.out‚26771.0888671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.out‚0 disk.sda3.out‚ 370.7265625 disk.dm-1.out‚ 370.7265625$ memory_percent‚90.80562591552734& proc.memory.availableMB‚ 1506.40625! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328* disk./.usagePercent‚17.408219711728144 disk.sda.out‚ 370.7265625 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 network.sent‚ 1.8784823e+07 disk.nvme0c0n1.in‚4.671875 disk.dm-1.in‚ 126.77734375 cpu‚25.33537670315346 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚26771.0888671875 disk.sda2.in‚0 disk.drbd2265.in‚0ÖŘÚí‚(:… ÌÕäÔ™®±~ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.37' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ‡vî‚(:ë ÌÕäÔ¼µ¢€ cpu‚25.334674330868634 proc.cpu.threads‚17 network.sent‚ 1.889375e+07 disk.sda1.in‚0 disk.drbd2080.in‚0" disk.dm-0.out‚26816.4130859375 disk.drbd2266.in‚0 disk.sda.out‚ 371.1328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0% disk./.usageGB‚259.41218185424805 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 371.1328125 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 network.recv‚ 2.332619e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875$ memory_percent‚90.80560207366943* disk./.usagePercent‚17.409369185556884 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.dm-1.out‚ 371.1328125! proc.memory.rssMB‚ 2693.203125' disk.nvme0c0n1.out‚26816.4130859375) proc.memory.availableMB‚ 1506.41015625 disk.nvme0n1.in‚4.671875 disk.sda3.in‚ 126.77734375% disk.nvme0n1.out‚26816.4130859375 disk.nvme0c0n1.in‚4.671875* proc.memory.percent‚16.438007354736328e’ï‚(:‡ ÛÕäÔèò–~ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÐzð‚(:ï ÛÕäÔ嘻€ network.sent‚ 1.8899063e+07% disk./.usageGB‚259.39025115966797 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875% disk.nvme0n1.out‚26865.9052734375) proc.memory.availableMB‚ 1506.16015625 disk.sda2.out‚0 network.recv‚ 2.335993e+06 disk.drbd2266.out‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-1.out‚ 371.63671875 disk.rbd1.out‚ 3.373046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0" disk.dm-0.out‚26865.9052734375 disk.drbd2266.in‚0$ memory_percent‚90.80712795257568* proc.memory.percent‚16.438007354736328* disk./.usagePercent‚17.407897398242536 disk.dm-0.in‚4.671875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.sda3.out‚ 371.63671875 cpu‚25.334044706864223 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 371.63671875' disk.nvme0c0n1.out‚26865.9052734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0! proc.memory.rssMB‚ 2693.2031251xªÕ2ñ- _wandbruntime‚6375  _runtime‚6375#*-ÇòÁ _step‚654 & lr-AdamW/pg1‚5.230571300744902e-05 & lr-AdamW/pg2‚5.230571300744902e-05 trainer/global_step‚7999 % _timestamp‚1.7884228757350655e+09 _runtime‚6375.472607085M$ªÌóÆ trainer/global_step‚7999 % _timestamp‚1.7884228757350655e+09 _runtime‚6375.472607085 _step‚654 & lr-AdamW/pg1‚5.230571300744902e-05 & lr-AdamW/pg2‚5.230571300744902e-05Ž­á 2ô- _wandbruntime‚6375  _runtime‚63752䬎õàO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116220266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6696254198784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954550 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507543583887872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459260718169000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471459228355072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788422875735841e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67808212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116220264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67808214 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6695597836288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471459228355072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452589768954880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514567987890688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464762956153344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116220266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464762956153344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597836288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6696254198784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150934512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7024591566848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514568175454720 _runtime‚6375.473622922 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38225299 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214120 trainer/global_step‚7999 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521¼ð“dë'öåO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚471459228355072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459261389646116 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464762974156288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚471458553989632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788422875735841e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67808212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464762974156288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116220264 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67808214 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6695597836288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚471459228355072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452589768954880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚514567987890688 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150954550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464762956153344 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116220266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464762956153344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6695597836288 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6696254198784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150934512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7024591566848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚471458553989632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚514568175454720 _runtime‚6375.473622922 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38225299 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6670949214120 trainer/global_step‚7999 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 267174816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452589785994240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116220266 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6696254198784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150954550 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚507543583887872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚459260718169000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414©Š8v2÷- _wandbruntime‚6376  _runtime‚6376ÛØñ&øëM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106046240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116234280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734524 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952966 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6696417264128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073639936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973030 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459317110211848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267207312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464819309192192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38229860 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚452644690124800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 % _timestamp‚1.7884228764718585e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67816382 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6671765649084 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073639936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507605234625536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471515726456320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6696417264128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106046676 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514630690182656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267187246 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952966 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7025298237952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400835072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116234282 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38230294 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6671765649084 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6697073639936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚507605234625536 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 267187246 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚471515726456320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6696417264128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106046676 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚514630690182656 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267187246 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150952966 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7025298237952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚471516400835072 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116234282 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464819327195136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116234280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38230294 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106046240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452644673085440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116234280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459316438734524 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150952966 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6696417264128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267207312 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6697073639936 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67816380 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150973030 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459317110211848 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267207312+~t2ú- _wandbruntime‚6376  _runtime‚6376^¸6XûÒ 0loss/FRt4/assignment_loss‚1.7584119555996913 2loss/SRqqt2/assignment_loss‚1.8539011796315512 % _timestamp‚1.7884228764739447e+09 _step‚657 2loss/SRqqt1/assignment_loss‚1.2162358994285265 .loss/FRt3/detection_loss‚2.360766173425046 1loss/SRqqt3/detection_loss‚1.7877058851774092 1loss/FBt4/assignment_loss‚0.15595406470129608 0loss/SRqqt1/detection_loss‚1.979761047826873 0loss/SRqqt2/detection_loss‚2.327559220790863 .loss/FBt4/detection_loss‚0.610659731497784 0loss/FBt3/assignment_loss‚0.0989313468337059 .loss/FRt2/detection_loss‚2.826890344128889 .loss/FBt3/detection_loss‚2.751767873764038 /loss/FRt3/assignment_loss‚2.826900518753312 trainer/global_step‚7999 _runtime‚6376.211392532 %loss/total_loss‚42.30891076635654 epoch‚6 /loss/FRt1/assignment_loss‚3.393601566553116 0loss/FRt2/assignment_loss‚3.3149299165781807 1loss/FBt2/assignment_loss‚0.22287566927546074 /loss/FRt1/detection_loss‚2.4294211907933154 /loss/FBt2/detection_loss‚2.2445307807488875 2loss/SRqqt3/assignment_loss‚1.6926981401112344 0loss/FBt1/assignment_loss‚0.3875710354460959 /loss/FRt4/detection_loss‚1.8789081795908071 1loss/SRqqt4/detection_loss‚1.0331719228870249 .loss/FBt1/detection_loss‚2.133636251091957 2loss/SRqqt4/assignment_loss‚1.0221208717214734Ò¢®]ü× 2loss/SRqqt1/assignment_loss‚1.2162358994285265 .loss/FRt3/detection_loss‚2.360766173425046 1loss/SRqqt3/detection_loss‚1.7877058851774092 1loss/FBt4/assignment_loss‚0.15595406470129608 0loss/SRqqt1/detection_loss‚1.979761047826873 0loss/SRqqt2/detection_loss‚2.327559220790863 .loss/FBt4/detection_loss‚0.610659731497784 0loss/FBt3/assignment_loss‚0.0989313468337059 .loss/FRt2/detection_loss‚2.826890344128889 .loss/FBt3/detection_loss‚2.751767873764038 /loss/FRt3/assignment_loss‚2.826900518753312 trainer/global_step‚7999 _runtime‚6376.211392532 %loss/total_loss‚42.30891076635654 epoch‚6 /loss/FRt1/assignment_loss‚3.393601566553116 0loss/FRt2/assignment_loss‚3.3149299165781807 1loss/FBt2/assignment_loss‚0.22287566927546074 /loss/FRt1/detection_loss‚2.4294211907933154 /loss/FBt2/detection_loss‚2.2445307807488875 2loss/SRqqt3/assignment_loss‚1.6926981401112344 0loss/FBt1/assignment_loss‚0.3875710354460959 /loss/FRt4/detection_loss‚1.8789081795908071 1loss/SRqqt4/detection_loss‚1.0331719228870249 .loss/FBt1/detection_loss‚2.133636251091957 2loss/SRqqt4/assignment_loss‚1.0221208717214734 0loss/FRt4/assignment_loss‚1.7584119555996913 2loss/SRqqt2/assignment_loss‚1.8539011796315512 % _timestamp‚1.7884228764739447e+09 _step‚657‘á7Ø’ý‚(:‡ êÕäԲ؋~ gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#XPôwþ‚(:ì êÕäÔÍôì€) disk./.usagePercent‚17.40888840340758$ disk./.usageGB‚259.4050178527832 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 network.recv‚ 2.340337e+06 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.373046875 disk.sda1.out‚0! proc.memory.rssMB‚ 2693.203125" disk.dm-0.out‚26901.0576171875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 372.12109375 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚4.671875 disk.drbd2266.out‚0 disk.drbd1716.in‚0% disk.nvme0n1.out‚26901.0576171875' disk.nvme0c0n1.out‚26901.0576171875 disk.sda2.out‚0 cpu‚25.33341648291885 disk.dm-1.out‚ 372.12109375 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375) proc.memory.availableMB‚ 1505.95703125 network.sent‚ 1.8905123e+07 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0$ memory_percent‚90.80836772918701* proc.memory.percent‚16.438007354736328 disk.sda.out‚ 372.12109375 disk.nvme0c0n1.in‚4.671875 disk.drbd2080.out‚05Ã’ÿ‚(:‡ ùÕäÔÙ€ž~ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>Kß.t€ ‚(:é ùÕäÔ짨€ disk.rbd1.out‚ 3.373046875( disk.nvme0c0n1.out‚26948.55517578125 disk.sda2.in‚0# memory_percent‚90.8083438873291( proc.memory.availableMB‚ 1505.9609375 network.sent‚ 1.9013758e+07 network.recv‚ 2.346334e+06 disk.dm-1.out‚ 372.484375 disk.sda3.out‚ 372.484375 disk.dm-0.in‚4.671875% disk./.usageGB‚259.41193771362305 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375) disk./.usagePercent‚17.40935280107469& disk.nvme0n1.out‚26948.55517578125 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2693.203125 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 372.484375 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚26948.55517578125* proc.memory.percent‚16.438007354736328 disk.dm-1.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.drbd2080.out‚0 cpu‚25.332759783838767 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.in‚0æEJ2 - _wandbruntime‚6414  _runtime‚6414rÙ‚øÉ‚ à trainer/global_step‚8049 % _timestamp‚1.7884229148390725e+09 _runtime‚6414.576608784 _step‚658 ' lr-AdamW/pg1‚5.1182758288989426e-05 ' lr-AdamW/pg2‚5.1182758288989426e-05ÁC΃ È trainer/global_step‚8049 % _timestamp‚1.7884229148390725e+09 _runtime‚6414.576608784 _step‚658 ' lr-AdamW/pg1‚5.1182758288989426e-05 ' lr-AdamW/pg2‚5.1182758288989426e-05’zÜ£•2„ - _wandbruntime‚6414  _runtime‚6414oLªç'… áO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880602 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151860564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106673073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517693798700544 % _timestamp‚1.7884229148399162e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474317866103296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510626096830976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151860564 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38456459 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106672614 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚6414.577745991 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866103296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6736583642112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38456002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116921066 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7067510111232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513524 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455334992522240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880602 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467580608095744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737240004608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455334975482880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517693606942208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462046760036408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7067677929472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚659T¼‘Xì'† æO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737240004608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151860564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68216612 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚510626120771072 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106673073 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68216614 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚517693798700544 % _timestamp‚1.7884229148399162e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚474317866103296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚510626096830976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151860564 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚8049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38456459 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚474317191737856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106672614 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚6414.577745991 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚474317866103296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6736583642112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6711784553528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38456002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116921066 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467580626098688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116921066 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7067510111232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462047431513524 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚455334992522240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467580626098688 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151880602 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467580608095744 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467580608095744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737240004608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚455334975482880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6712438991284 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚517693606942208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚462046760036408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚474317191737856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7067677929472 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 _step‚659 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151880602 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521“Þ ZÍ2‡ - _wandbruntime‚6415  _runtime‚6415ÿXÞˆ ëM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462102480601932 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7068532072960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151899082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7068366365184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6713255426456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467636979137536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106685803 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517756303581696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0°ôÖ XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106685347 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38461021 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚660 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517756113933824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467636961134592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68224780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 % _timestamp‚1.7884229155741205e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6415.312283184ó©ö&‰ ðM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116935080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚455389879613440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚510687747568640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268814098 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚517756113933824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚455389896652800 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8049 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467636961134592 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268834164 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68224780 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462103152079256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268834164 % _timestamp‚1.7884229155741205e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151879018 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚6415.312283184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚462102480601932 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151899082 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7068532072960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚474375038583296 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6737403069952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚474374364204544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151879018 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151899082 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7068366365184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6713255426456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467636979137536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚510687771508736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106685803 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116935082 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚517756303581696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6712600988492 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106685347 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38461021 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚660 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8099 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458080182010880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6753274330692 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117621866 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513708657654272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477176503851520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520819353334272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6453.716517088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117621864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7110695680000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107310349 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152786616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477176503851520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7110534153216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6778225810432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107310811 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38685337 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38685797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806654 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470398278041088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080[çÆì'˜ æO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117621864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6778225810432 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚470398278041088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚458080199050240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6752619892936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477175829486080 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477175829486080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚8099 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458080182010880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6753274330692 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117621866 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270428520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚513708657654272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚477176503851520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520819353334272 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6453.716517088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117621864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270408480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7110695680000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6777569447936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107310349 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152786616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚477176503851520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6777569447936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7110534153216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚464832801903816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6778225810432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152806654 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107310811 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38685337 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚513708633714176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68625014 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38685797 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚464833473380932 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152806654 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚470398278041088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚470398260038144 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464889193946664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107323086 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805070 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8099 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6753436327900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107323552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045251584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884229547267344e+09 _step‚664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117635882 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68633180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513770308391936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6454.464779621 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513770284451840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68633182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470454631079936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7111383093760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676331520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520881667545600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7111552995840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477233001952768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881861387776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388875776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38690370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464888522469340 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430$þgö&› ðM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚477233676331520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚520881667545600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7111552995840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚477233001952768 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152825134 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520881861387776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6779045251584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458135103180800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6778388875776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270440950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38690370 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270461016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464888522469340 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38689906 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464889193946664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107323086 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚477233676331520 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117635882 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117635880 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117635880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152805070 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8099 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152825134 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6753436327900 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6778388875776 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107323552 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6779045251584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6754090765864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270461016 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 % _timestamp‚1.7884229547267344e+09 _step‚664 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117635882 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152805070 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68633180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚513770308391936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚458135086141440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚466 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚6454.464779621 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470454631079936 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚513770284451840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68633182 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470454631079936 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7111383093760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270440950 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808˜¦¬nÚ2œ - _wandbruntime‚6454  _runtime‚6454çBÔ… Ô /loss/FRt2/assignment_loss‚3.696876335889101 2loss/SRqqt2/assignment_loss‚1.7757842736318707 2loss/SRqqt3/assignment_loss‚1.6731737787362004 2loss/FBt3/assignment_loss‚0.014445722102910471 1loss/FBt1/assignment_loss‚0.18672229934600182 .loss/FBt1/detection_loss‚2.061767026782036 /loss/FRt1/assignment_loss‚3.532344550436193 1loss/FBt2/assignment_loss‚0.16521468968118874 /loss/FRt4/detection_loss‚1.6458744332939386 1loss/SRqqt2/detection_loss‚2.2695541232824326 .loss/SRqqt3/detection_loss‚1.5578292873001 -loss/FBt2/detection_loss‚2.57803635937827 0loss/FBt4/detection_loss‚0.48549295773846096 % _timestamp‚1.7884229547284493e+09 /loss/FRt1/detection_loss‚2.0822263425046748 _runtime‚6454.465940312 0loss/FRt4/assignment_loss‚1.0690232348442077 .loss/FRt2/detection_loss‚2.858962967991829 .loss/FRt3/detection_loss‚øè8RÕ2.132082045376301 1loss/SRqqt1/detection_loss‚1.8658724475738615 1loss/SRqqt4/detection_loss‚1.0540855519053802 1loss/FBt4/assignment_loss‚0.11713416368355967 /loss/FBt3/detection_loss‚3.9681933720906577 2loss/SRqqt1/assignment_loss‚1.4058130310718404 1loss/SRqqt4/assignment_loss‚1.046687996880689 &loss/total_loss‚41.889658011037696 epoch‚6 trainer/global_step‚8099 _step‚665 /loss/FRt3/assignment_loss‚2.646461019515991cûþ_ž Ù /loss/FRt1/assignment_loss‚3.532344550436193 1loss/FBt2/assignment_loss‚0.16521468968118874 /loss/FRt4/detection_loss‚1.6458744332939386 1loss/SRqqt2/detection_loss‚2.2695541232824326 .loss/SRqqt3/detection_loss‚1.5578292873001 -loss/FBt2/detection_loss‚2.57803635937827 0loss/FBt4/detection_loss‚0.48549295773846096 % _timestamp‚1.7884229547284493e+09 /loss/FRt1/detection_loss‚2.0822263425046748 _runtime‚6454.465940312 0loss/FRt4/assignment_loss‚1.0690232348442077 .loss/FRt2/detection_loss‚2.858962967991829 .loss/FRt3/detection_loss‚2.132082045376301 1loss/SRqqt1/detection_loss‚1.8658724475738615 1loss/SRqqt4/detection_loss‚1.0540855519053802 1loss/FBt4/assignment_loss‚0.11713416368355967 /loss/FBt3/detection_loss‚3.9681933720906577 2loss/SRqqt1/assignment_loss‚1.4058130310718404 1loss/SRqqt4/assignment_loss‚1.046687996880689 &loss/total_loss‚41.889658011037696 epoch‚6 trainer/global_step‚8099 _step‚665 /loss/FRt3/assignment_loss‚2.646461019515991 /loss/FRt2/assignment_loss‚3.696876335889101 2loss/SRqqt2/assignment_loss‚1.7757842736318707 2loss/SRqqt3/assignment_loss‚1.6731737787362004 2loss/FBt3/assignment_loss‚0.014445722102910471 1loss/FBt1/assignment_loss‚0.18672229934600182 .loss/FBt1/detection_loss‚2.061767026782036™c'·†‘Ÿ ‚(:† µÖäÔ¹è¥~ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?˜Å{  ‚(:ð µÖäÔÞõÉ proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda3.out‚ 374.87109375 disk.dm-1.out‚ 374.87109375 cpu‚25.330174737256296 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875 disk.sda1.in‚0& disk.nvme0n1.out‚27102.34521484375 disk.rbd0.out‚0( proc.memory.availableMB‚ 1505.8671875 disk.sda.out‚ 374.87109375 disk.sda1.out‚0 network.sent‚ 1.9139167e+07$ disk./.usageGB‚259.4199676513672( disk.nvme0c0n1.out‚27102.34521484375 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.373046875! proc.memory.rssMB‚ 2693.203125 disk.drbd2266.in‚0 disk.drbd1716.out‚0$ memory_percent‚90.80891609191895* disk./.usagePercent‚17.409891696934267 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.sda2.in‚0# disk.dm-0.out‚27102.34521484375 disk.rbd0.in‚0 network.recv‚ 2.364735e+06 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.438007354736328§˜ÌÕ’¡ ‚(:‡ ÄÖäÔ¿©‘~ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.732' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.5464 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦UÆss¢ ‚(:è ÄÖäÔù¥“€ disk.sda3.in‚ 126.77734375# disk.dm-0.out‚27137.07958984375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0$ memory_percent‚90.81039428710938 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚27137.07958984375 disk.drbd2265.out‚0 disk.sda.out‚ 375.3984375 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚27137.07958984375 proc.cpu.threads‚17 disk.rbd1.out‚ 3.373046875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328 disk.dm-1.out‚ 375.3984375 network.recv‚ 2.371504e+06$ disk./.usageGB‚259.4261245727539 disk.rbd1.in‚0 cpu‚25.329540041800264) disk./.usagePercent‚17.41030489309453 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 1.9248115e+07 disk.sda3.out‚ 375.3984375 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0$ proc.memory.availableMB‚1505.625tÌ÷2£ - _wandbruntime‚6493  _runtime‚6493À¯ïGǤ Á & lr-AdamW/pg2‚4.895781325768093e-05 trainer/global_step‚8149 % _timestamp‚1.7884229932866433e+09 _runtime‚6493.024184116 _step‚666 & lr-AdamW/pg1‚4.895781325768093e-05>Ó›¦Ì¥ Æ _step‚666 & lr-AdamW/pg1‚4.895781325768093e-05 & lr-AdamW/pg2‚4.895781325768093e-05 trainer/global_step‚8149 % _timestamp‚1.7884229932866433e+09 _runtime‚6493.024184116šèŠg(2¦ - _wandbruntime‚6493  _runtime‚6493ŠËÓfæ'§ àO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38915100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6493.025156442 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153643078144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118322666 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚667 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153817187840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6793455232344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523944813675520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523945011725312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153712668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107948514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153712668 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272035332 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440º kë'¨ åO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚473215911980544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153817187840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6819211616256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚480035141599744 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153732706 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6793455232344 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6819211616256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚523944813675520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118322666 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚523945011725312 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153712668 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚473215911980544 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480034467234304 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107948514 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467619515248340 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153712668 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6818555253760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460825405578240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272035332 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118322664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6794109670100 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38915100 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _runtime‚6493.025156442 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153643078144 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118322666 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚667› &ê“2© - _wandbruntime‚6493  _runtime‚6493\hÒ…_ª ëM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69041582 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751186 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272067802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314079744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524007522045952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819374681600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚668 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031057408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107960760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524007342883840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7154521548800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6493.783931403 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460880309708800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731122 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884229940457695e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751186 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38919180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactiv8sw›’ e_split_bytes.all.current‚ 179162112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118336682 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473272265019392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8149 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272087868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚668 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6820031057408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚473272283022336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107960760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524007342883840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7154521548800 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6493.783931403 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118336680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚480092314079744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460880309708800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153731122 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884229940457695e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751186 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6794926105272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚473272283022336 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6794271667308 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38919180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272087868 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272067802 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118336682 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚473272265019392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8149 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6819374681600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚516852821335040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153731122 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751186 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272067802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6820031057408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480092314079744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467675235814072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467674564336748 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118336682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69041580 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524007522045952 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6819374681600œ‘¿AL2¬ - _wandbruntime‚6493  _runtime‚6493¯Ìgä\­ Ö /loss/FBt1/detection_loss‚1.8923136666417122 /loss/FRt2/assignment_loss‚3.343271616846323 1loss/SRqqt1/assignment_loss‚1.639303448338372 %loss/total_loss‚42.35831181733971 1loss/SRqqt3/detection_loss‚1.7227271643395607 epoch‚6 0loss/FRt1/assignment_loss‚3.0980035468935965 2loss/SRqqt4/assignment_loss‚0.9584027023689039 1loss/FBt2/assignment_loss‚0.10782215171704995 .loss/FRt3/detection_loss‚2.439835045053515 0loss/FRt4/assignment_loss‚1.5017305282836264 1loss/SRqqt1/detection_loss‚1.7720231047521036 /loss/FBt4/detection_loss‚0.5533630594938665 trainer/global_step‚8149 1loss/FBt1/assignment_loss‚0.45179489014359814 .loss/FRt2/detection_loss‚2.867188109084964 .loss/FBt2/detection_loss‚2.776016555049203 /loss/FRt3/assignment_loss‚2.863498780616494 2loss/SRqqt3/assignment_loss‚1.5369972739225397 /loss/FRt4/detection_loss‚1.7838465500068157 % _timestamp‚1.7884229940479891e+09 2loss/SRqqt2/assignment_loss‚2.1655729233190932 1loss/SRqqt2/detection_loss‚2.3941721325290612 /loss/FBt3/detection_loss‚2.7127123143937855 _runtime‚6493.785463354 _step‚669 0loss/FBt3/assignment_loss‚0.4130913567625814 1loss/FBt4/assignment_loss‚0.11243516323061724 /loss/FRt1/detection_loss‚2.3424369409680366 1loss/SRqqt4/detection_loss‚0.9097527925842914ÊÁa® Û 2loss/SRqqt2/assignment_loss‚2.1655729233190932 1loss/SRqqt2/detection_loss‚2.3941721325290612 /loss/FBt3/detection_loss‚2.7127123143937855 _runtime‚6493.785463354 _step‚669 0loss/FBt3/assignment_loss‚0.4130913567625814 1loss/FBt4/assignment_loss‚0.11243516323061724 /loss/FRt1/detection_loss‚2.3424369409680366 1loss/SRqqt4/detection_loss‚0.9097527925842914 /loss/FBt1/detection_loss‚1.8923136666417122 /loss/FRt2/assignment_loss‚3.343271616846323 1loss/SRqqt1/assignment_loss‚1.639303448338372 %loss/total_loss‚42.35831181733971 1loss/SRqqt3/detection_loss‚1.7227271643395607 epoch‚6 0loss/FRt1/assignment_loss‚3.0980035468935965 2loss/SRqqt4/assignment_loss‚0.9584027023689039 1loss/FBt2/assignment_loss‚0.10782215171704995 .loss/FRt3/detection_loss‚2.439835045053515 0loss/FRt4/assignment_loss‚1.5017305282836264 1loss/SRqqt1/detection_loss‚1.7720231047521036 /loss/FBt4/detection_loss‚0.5533630594938665 trainer/global_step‚8149 1loss/FBt1/assignment_loss‚0.45179489014359814 .loss/FRt2/detection_loss‚2.867188109084964 .loss/FBt2/detection_loss‚2.776016555049203 /loss/FRt3/assignment_loss‚2.863498780616494 2loss/SRqqt3/assignment_loss‚1.5369972739225397 /loss/FRt4/detection_loss‚1.7838465500068157 % _timestamp‚1.7884229940479891e+09¨ŒT¯ ‚(:’ ÓÖäÔÈ”˜~ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚140.837' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.167399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦î|° ‚(:ñ ÓÖäÔÐÕ¤* disk./.usagePercent‚17.407897398242536 disk.dm-1.in‚ 126.77734375 disk.sda2.out‚0 disk.drbd2265.in‚0% disk./.usageGB‚259.39025115966797 disk.rbd1.out‚ 3.373046875( disk.nvme0c0n1.out‚27179.06005859375 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 network.recv‚ 2.375097e+06 disk.sda3.out‚ 376.08984375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875( proc.memory.availableMB‚ 1505.8359375! proc.memory.rssMB‚ 2693.203125 disk.drbd1716.in‚0 disk.sda2.in‚0$ memory_percent‚90.80910682678223 disk.sda.out‚ 376.08984375 network.sent‚ 1.9253441e+07 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 376.08984375 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚27179.06005859375 disk.rbd0.in‚0# disk.dm-0.out‚27179.06005859375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.438007354736328 cpu‚25.328887913740296 disk.sda1.in‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.out‚0xðH œ± ‚(:‘ âÖäÔ³õŽ~ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.91' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.382000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¨ t{² ‚(:ð âÖäÔ貓 disk.drbd1716.in‚0$ memory_percent‚90.80910682678223 proc.cpu.threads‚17 network.sent‚ 1.9361939e+07 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875% disk./.usageGB‚259.42918395996094 disk.sda2.in‚0 disk.sda3.out‚ 376.57421875 disk.rbd1.out‚ 3.373046875 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0* proc.memory.percent‚16.438007354736328 network.recv‚ 2.381199e+06 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2693.203125 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 376.57421875 disk.sda.out‚ 376.57421875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚27214.38037109375( proc.memory.availableMB‚ 1505.8359375 disk.drbd2265.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚27214.38037109375 cpu‚25.328248972844357) disk./.usagePercent‚17.41051021113699 disk.drbd2080.in‚0# disk.dm-0.out‚27214.38037109375T¢‡V³ ‚(:’ ñÖäÔŘ²~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Mqs´ ‚(:è ñÖäԇ؀ disk.dm-0.in‚4.671875# disk.dm-0.out‚27257.15966796875 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 377.140625 network.recv‚ 2.385598e+06 disk.sda2.in‚0$ memory_percent‚90.80910682678223* proc.memory.percent‚16.438007354736328 network.sent‚ 1.9367989e+07( disk.nvme0c0n1.out‚27257.15966796875 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 377.140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚27257.15966796875( proc.memory.availableMB‚ 1505.8359375! proc.memory.rssMB‚ 2693.203125 cpu‚25.327616890084474 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 377.140625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0( disk./.usagePercent‚17.4113499158493$ disk./.usageGB‚259.4416961669922 disk.sda2.out‚0 disk.nvme0n1.in‚4.671875 proc.cpu.threads‚177زó2µ - _wandbruntime‚6531  _runtime‚6531·^4 Ç¶ Á % _timestamp‚1.7884230322200441e+09 _runtime‚6531.957579422 _step‚670 & lr-AdamW/pg1‚4.785628662004169e-05 & lr-AdamW/pg2‚4.785628662004169e-05 trainer/global_step‚8199îÀÔÌ· Æ & lr-AdamW/pg1‚4.785628662004169e-05 & lr-AdamW/pg2‚4.785628662004169e-05 trainer/global_step‚8199 % _timestamp‚1.7884230322200441e+09 _runtime‚6531.957579422 _step‚670žµ,þ2¸ - _wandbruntime‚6531  _runtime‚65314y¸»æ'¹ àO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 273662184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6860197422080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 _runtime‚6531.958591583 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119023464 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154658758 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7196695333376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119023466 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚482893104982528 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119023464 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39144707 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚463570595066880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39144253 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6834945009508 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788423032220802e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527070402813952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6859541059584 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519873707480576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108586065 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚8199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476033581925888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108586521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚671 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119023466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154638720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6859541059584 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154638720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚482893779347968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6860197422080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚527070402813952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚463570612106240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6859541059584 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚482893104982528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69441812 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚519873707480576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚482893779347968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108586065 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚8199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚476033581925888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108586521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6834290571752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚671 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚476033563922944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69441814 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚519873731420672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30Ÿ±GÇñ2» - _wandbruntime‚6532  _runtime‚6532äsð&¼ êM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7197549664256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69449982 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39149174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119037480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108598712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚8199 _runtime‚6532.709260488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6861016863232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6861016863232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273694654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470460606204156 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7197717469184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360487424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 $ _timestamp‚1.788423032971232e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108599156 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430¬)_dõ&½ ïM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚463625516236800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119037482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚519935382158336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚519935358218240 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108599156 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7197549664256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154657174 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482950951827968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69449980 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69449982 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚476089916961792 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39149174 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6860360487424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119037480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108598712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚8199 _runtime‚6532.709260488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273714720 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6861016863232 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚476089934964736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6861016863232 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273694654 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚470460606204156 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119037482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7197717469184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚476089934964736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6860360487424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6835761444680 $ _timestamp‚1.788423032971232e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154677238 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273714720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚463625499197440 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 -Þl.2¾ - _wandbruntime‚6532  _runtime‚6532%¨U¿ Ï /loss/FRt1/detection_loss‚2.2826497638836885 .loss/FRt3/detection_loss‚2.544990753516173 1loss/SRqqt3/detection_loss‚1.7043287951727302 epoch‚6 2loss/SRqqt3/assignment_loss‚1.2397983666433363 0loss/FBt1/assignment_loss‚0.3359327619481418 -loss/FBt3/detection_loss‚2.96066465228796 /loss/FRt1/assignment_loss‚3.394253897361266 2loss/SRqqt2/assignment_loss‚1.6941533002715845 .loss/FRt4/detection_loss‚1.935172297592674 trainer/global_step‚8199 _step‚673 2loss/SRqqt1/assignment_loss‚1.2895423845977199 1loss/FBt3/assignment_loss‚0.18781754082078805 1loss/SRqqt4/detection_loss‚0.9634530481628396 /loss/FBt1/detection_loss‚2.5149432751867504 % _timestamp‚1.7884230329732957e+09 .loss/FBt2/assignment_loss‚1.04777577288233 0loss/FBt4/assignment_loss‚0.2643305486941501 1loss/SRqqt1/detection_loss‚1.6318480375469948 .loss/FBt2/detection_loss‚2.897391046796526 0loss/FRt3/assignment_loss‚2.6206551515139065 0loss/FRt4/assignment_loss‚1.5940436015323716 .loss/FRt2/detection_loss‚2.914675167922316 1loss/SRqqt2/detection_loss‚2.6070015843097982 -loss/FBt4/detection_loss‚0.69534656527685 _runtime‚6532.710764237 2loss/SRqqt4/assignment_loss‚0.9061843543525108 &loss/total_loss‚43.619516993770475 /loss/FRt2/assignment_loss‚3.392564325497068O«ø]ZÀ Ô -loss/FBt3/detection_loss‚2.96066465228796 /loss/FRt1/assignment_loss‚3.394253897361266 2loss/SRqqt2/assignment_loss‚1.6941533002715845 .loss/FRt4/detection_loss‚1.935172297592674 trainer/global_step‚8199 _step‚673 2loss/SRqqt1/assignment_loss‚1.2895423845977199 1loss/FBt3/assignment_loss‚0.18781754082078805 1loss/SRqqt4/detection_loss‚0.9634530481628396 /loss/FBt1/detection_loss‚2.5149432751867504 % _timestamp‚1.7884230329732957e+09 .loss/FBt2/assignment_loss‚1.04777577288233 0loss/FBt4/assignment_loss‚0.2643305486941501 1loss/SRqqt1/detection_loss‚1.6318480375469948 .loss/FBt2/detection_loss‚2.897391046796526 0loss/FRt3/assignment_loss‚2.6206551515139065 0loss/FRt4/assignment_loss‚1.5940436015323716 .loss/FRt2/detection_loss‚2.914675167922316 1loss/SRqqt2/detection_loss‚2.6070015843097982 -loss/FBt4/detection_loss‚0.69534656527685 _runtime‚6532.710764237 2loss/SRqqt4/assignment_loss‚0.9061843543525108 &loss/total_loss‚43.619516993770475 /loss/FRt2/assignment_loss‚3.392564325497068 /loss/FRt1/detection_loss‚2.2826497638836885 .loss/FRt3/detection_loss‚2.544990753516173 1loss/SRqqt3/detection_loss‚1.7043287951727302 epoch‚6 2loss/SRqqt3/assignment_loss‚1.2397983666433363 0loss/FBt1/assignment_loss‚0.3359327619481418¡²K8à’Á ‚(:‡ €×äÔÏâ®~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï|A ‚(:î €×äÔ°™ì€ disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 cpu‚25.326980067919486( disk.nvme0c0n1.out‚27292.25341796875 disk.drbd2266.in‚0 disk.sda.out‚ 377.8828125! proc.memory.rssMB‚ 2693.203125 network.recv‚ 2.390307e+06 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚27292.25341796875) proc.memory.availableMB‚ 1505.80859375* disk./.usagePercent‚17.412085169487636 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sd±¾'ß8a3.out‚ 377.8828125 disk.dm-1.out‚ 377.8828125* proc.memory.percent‚16.438007354736328 proc.cpu.threads‚17% disk./.usageGB‚259.45265197753906 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚27292.25341796875 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.671875# memory_percent‚90.8092737197876 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 3.373046875 network.sent‚ 1.9374306e+07 disk.rbd0.out‚0 disk.drbd2265.in‚0éƒ ’à ‚(:‡ ×äÔòºö} gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š’¾ázÄ ‚(:ï ×äÔ¼úø€$ disk./.usageGB‚259.4658546447754 disk.dm-0.in‚4.671875 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚ 2.396023e+06 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17# disk.dm-0.out‚27325.33056640625 disk.sda.out‚ 378.26171875 disk.drbd1716.out‚0& disk.nvme0n1.out‚27325.33056640625 disk.nvme0c0n1.in‚4.671875! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328* disk./.usagePercent‚17.412971211563644 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda3.out‚ 378.26171875$ memory_percent‚90.80805778503418( proc.memory.availableMB‚ 1506.0078125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 378.26171875 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚27325.33056640625 disk.drbd1716.in‚0 cpu‚25.326310083272066 network.sent‚ 1.948223e+07 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0.kõ¯Å ‚(:… ž×äÔúþª~ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚159.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.982 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Æ«œ|Æ ‚(:ñ ž×äÔ¿ËÀ disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 378.94921875# disk.dm-0.out‚27360.38525390625 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.373046875 disk.drbd2265.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875* disk./.usagePercent‚17.414189295411557 disk.drbd2265.out‚0 disk.drbd2080.out‚0& disk.nvme0n1.out‚27360.38525390625! proc.memory.rssMB‚ 2693.203125 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 378.94921875 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚4.671875 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 378.94921875 disk.sda1.in‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.438007354736328% disk./.usageGB‚259.48400497436523 disk.sda1.out‚0 cpu‚25.325722827766942 network.sent‚ 1.9488231e+07 disk.drbd2266.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚27360.38525390625$ memory_percent‚90.80948829650879( proc.memory.availableMB‚ 1505.7734375 network.recv‚ 2.400518e+06 disk.sda2.out‚0 proc.cpu.threads‚17X;=2Ç - _wandbruntime‚6571  _runtime‚6571~5…\ÈÈ Â ' lr-AdamW/pg1‚4.6762363822009355e-05 ' lr-AdamW/pg2‚4.6762363822009355e-05 trainer/global_step‚8249 $ _timestamp‚1.788423071336098e+09 _runtime‚6571.073644191 _step‚674wݬbÍÉ Ç ' lr-AdamW/pg1‚4.6762363822009355e-05 ' lr-AdamW/pg2‚4.6762363822009355e-05 trainer/global_step‚8249 $ _timestamp‚1.788423071336098e+09 _runtime‚6571.073644191 _step‚674¢œ¬wC2Ê - _wandbruntime‚6571  _runtime‚6571Ÿ´Äæ'Ë àO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155584810 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522956268303872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155564772 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109223890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788423071337072e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119724266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6901183227904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526865408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485752417096192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7239751529472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6900526865408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39373678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239929833472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183227904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485751742730752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _step‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473191598983156 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚466315801594880 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109224363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530196198137344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195995893248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522956244363776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚8249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚6571.075021892 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076)CÖgë'Ì åO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 275309076 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155584810 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚478851215865344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚522956268303872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155564772 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109223890 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚466315818634240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 $ _timestamp‚1.788423071337072e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119724266 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6901183227904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178304000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6900526865408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69850212 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚485752417096192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7239751529472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6875125911160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6900526865408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚473 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485751742730752 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39373678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239929833472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6901183227904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485751742730752 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155584810 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 _step‚675 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚473191598983156 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚478851233868288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 202244096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚466315801594880 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119724266 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109224363 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚530196198137344 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚530195995893248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚522956244363776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 trainer/global_step‚8249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚6571.075021892£gmY2Í - _wandbruntime‚6571  _runtime‚6571³YïYÎ ëM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6901346293248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478907586907136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 % _timestamp‚1.7884230720603335e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346293248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321506 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109236560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7240773673472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 _runtime‚6571.798345624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530258500969984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚676 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39378180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39378653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_tì7’^˜rain_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚8249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119738282 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485808915197440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002669056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ªËñÚö&Ï ðM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69858380 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚8249 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155603290 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119738282 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚485808915197440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6902002669056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6901346293248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚478907586907136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275341572 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 275321506 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155603290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 % _timestamp‚1.7884230720603335e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6901346293248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119738280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69858382 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876596784088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275321506 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚478907586907136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚466370722764800 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109236560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚473 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7240773673472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚523017895101440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 _runtime‚6571.798345624 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119738280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚530258500969984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚478907568904192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚475 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚676 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚473247319548888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39378180 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39378653 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275341572 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6916615688324 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481668867807744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469061008122880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7282814598144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156490824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚481668885810688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533321595845120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120425064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533321783409152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6609.517477072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39603198 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488610380478976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6916615688324 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚481668867807744 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70258614 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚469061008122880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488611054844416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120425066 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7282814598144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120425066 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156490824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _step‚679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pooOs™LÆl.allocated‚481668885810688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚533321595845120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8299 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚481668885810688 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120425064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6941512671232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚481668867807744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70258612 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚533321783409152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚6609.517477072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120425064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39603198 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509278 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6942332099072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39608303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7283672703488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚8299 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39607838 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884231107393322e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6942332099072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109874618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6610.477686799 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7283836314112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533384292238848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376xè‹¿ö&ß ðM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488668227324416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488667552945664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120439080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916777685532 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120439082 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120439080 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156509278 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6942332099072 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156509278 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6917432123496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39608303 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚467 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7283672703488 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚469115912253440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚8299 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276968424 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6942988474880 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚481725220846592 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39607838 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚481725238849536 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚526100455924736 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚476033361416296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 % _timestamp‚1.7884231107393322e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6942332099072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6942988474880 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109874618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚469115929292800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6610.477686799 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7283836314112 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚533384292238848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156529342 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156529342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚488668227324416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276968424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚526100431984640 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276948358 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚476032689938972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276948358 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154839552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788423148990401e+09 _step‚683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536447369510912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110500406 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436914 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7326027440640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325868011008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6982498477056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491469692592640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498477056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542740 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154839552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39832941 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0c­Xóë'î åO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70667012 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6982498477056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 278542740 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121125864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6983154839552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121125864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚491469692592640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39832941 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚491469018227200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6983154839552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 $ _timestamp‚1.788423148990401e+09 _step‚683 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536447369510912 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110500406 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157436914 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121125866 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7326027440640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7325868011008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6982498477056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚529121342070272 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121125866 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚471806214650880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70667014 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 157416876 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484486537753088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚491469692592640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278562780 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚484486519750144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚471806231690240 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680«ߎ2ï - _wandbruntime‚6649  _runtime‚6649² 6áñ&ð ëM _runtime‚6649.456186251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7326747176448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884231497181041e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326902398464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484542872788992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39837480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157435330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110513114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471861118781440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8349 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110512660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509895206400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39837932 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471861135820800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278575210 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536509716044288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0o§Â"œñ ðM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110512660 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6958267462904 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 157435330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70675180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚536509895206400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39837932 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚471861135820800 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚491526865072640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚491526865072640 M>DeviceStatsMonitor.on_train_batch_end/alloš)À}Z#cated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚491526190693888 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121139882 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478819403283704 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278575210 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536509716044288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121139880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121139880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6649.456186251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6983317904896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7326747176448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚452 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 % _timestamp‚1.7884231497181041e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121139882 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7326902398464 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278575210 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484542890791936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278595276 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚484542872788992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484542872788992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6957613024940 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278595276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39837480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6983974280704 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 157435330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70675182 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110513114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 157455394 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚471861118781440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8349¬ý?,h2ò - _wandbruntime‚6649  _runtime‚6649‹|îIZó Ô /loss/FBt4/detection_loss‚0.5993472559536558 0loss/FBt4/assignment_loss‚0.1667102778340191 /loss/FBt3/detection_loss‚2.0287311747670174 _runtime‚ 6649.45747305 0loss/FRt1/assignment_loss‚3.1425943306514195 /loss/FRt4/assignment_loss‚1.308157902293735 2loss/SRqqt2/assignment_loss‚2.2306710296207006 0loss/FBt1/assignment_loss‚0.3321566542144865 1loss/SRqqt3/detection_loss‚1.5644427953795954 /loss/FBt1/detection_loss‚1.9395447000861168 2loss/SRqqt1/assignment_loss‚1.2185380282097085 0loss/FBt2/assignment_loss‚0.3311561729758978 epoch‚6 trainer/global_step‚8349 _step‚685 1loss/SRqqt4/assignment_loss‚0.884675906320447 /loss/FRt4/detection_loss‚2.0803452637460498 /loss/FBt2/detection_loss‚3.0935381174087526 /loss/FRt1/detection_loss‚2.5857001304626466 .loss/FRt2/detection_loss‚2.947138943274816 /loss/FRt3/detection_loss‚2.3601107948149243 0loss/SRqqt2/detection_loss‚2.104538568523195 0loss/SRqqt4/detection_loss‚0.861840174112418 0loss/FRt2/assignment_loss‚3.5036628882090253 2loss/SRqqt3/assignment_loss‚1.5559225227683782 0loss/FBt3/assignment_loss‚0.3018224697734695 &loss/total_loss‚42.206484715220085 0loss/FRt3/assignment_loss‚3.1698900933067002 1loss/SRqqt1/detection_loss‚1.8952485205129135 % _timestamp‚1.7884231497200003e+09þç’:_ô Ù 0loss/FRt3/assignment_loss‚3.1698900933067002 1loss/SRqqt1/detection_loss‚1.8952485205129135 % _timestamp‚1.7884231497200003e+09 /loss/FBt4/detection_loss‚0.5993472559536558 0loss/FBt4/assignment_loss‚0.1667102778340191 /loss/FBt3/detection_loss‚2.0287311747670174 _runtime‚ 6649.45747305 0loss/FRt1/assignment_loss‚3.1425943306514195 /loss/FRt4/assignment_loss‚1.308157902293735 2loss/SRqqt2/assignment_loss‚2.2306710296207006 0loss/FBt1/assignment_loss‚0.3321566542144865 1loss/SRqqt3/detection_loss‚1.5644427953795954 /loss/FBt1/detection_loss‚1.9395447000861168 2loss/SRqqt1/assignment_loss‚1.2185380282097085 0loss/FBt2/assignment_loss‚0.3311561729758978 epoch‚6 trainer/global_step‚8349 _step‚685 1loss/SRqqt4/assignment_loss‚0.884675906320447 /loss/FRt4/detection_loss‚2.0803452637460498 /loss/FBt2/detection_loss‚3.0935381174087526 /loss/FRt1/detection_loss‚2.5857001304626466 .loss/FRt2/detection_loss‚2.947138943274816 /loss/FRt3/detection_loss‚2.3601107948149243 0loss/SRqqt2/detection_loss‚2.104538568523195 0loss/SRqqt4/detection_loss‚0.861840174112418 0loss/FRt2/assignment_loss‚3.5036628882090253 2loss/SRqqt3/assignment_loss‚1.5559225227683782 0loss/FBt3/assignment_loss‚0.3018224697734695 &loss/total_loss‚42.206484715220085­Ü.¨‘õ ‚(:† ø×äÔ§¾¨~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.411 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ÍT`{ö ‚(:ð ø×äÔ—€ƒ& disk.nvme0n1.out‚27597.89404296875 disk.rbd0.in‚0 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0$ memory_percent‚90.81025123596191! proc.memory.rssMB‚ 2693.203125 proc.cpu.threads‚17 network.recv‚ 2.429783e+06 disk.sda.out‚ 382.07421875 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.373046875 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.drbd2080.out‚0# disk.dm-0.out‚27597.89404296875 disk.dm-1.out‚ 382.07421875 network.sent‚ 1.9729205e+07% disk./.usageGB‚259.49317169189453( disk.nvme0c0n1.out‚27597.89404296875 disk.sda3.out‚ 382.07421875 disk.drbd2265.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.438007354736328 disk.sda1.in‚0 disk.dm-1.in‚ 126.77734375( proc.memory.availableMB‚ 1505.6484375* disk./.usagePercent‚17.414804481516338 disk.sda.in‚ 126.77734375 disk.sda2.out‚0 cpu‚25.321970236004213»E°÷ ‚(:’ ‡ØäÔ£æ¸~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.405' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.480999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿L%±qø ‚(:æ ‡ØäÔ’˜½€ disk.sda1.out‚0 disk.sda3.out‚ 382.578125 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875 disk.nvme0c0n1.in‚4.671875* disk./.usagePercent‚17.418503534378576% disk./.usageGB‚259.54829025268555 disk.sda.in‚ 126.77734375! proc.memory.rssMB‚ 2693.203125* proc.memory.percent‚16.438007354736328 cpu‚25.321402937267422% disk.nvme0n1.out‚27637.0615234375 disk.drbd2080.in‚0 disk.rbd1.in‚0 network.recv‚ 2.43514e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0" disk.dm-0.out‚27637.0615234375 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17 disk.nvme0n1.in‚4.671875 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 382.578125$ memory_percent‚90.81029891967773' proc.memory.availableMB‚ 1505.640625 network.sent‚ 1.9837206e+07 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚27637.0615234375 disk.sda2.in‚0 disk.dm-1.out‚ 382.578125 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375†lŒ 2ù - _wandbruntime‚6688  _runtime‚6688v=«aÇú Á trainer/global_step‚8399 % _timestamp‚1.7884231882839286e+09 _runtime‚6688.021469558 _step‚686 & lr-AdamW/pg1‚4.352848997641106e-05 & lr-AdamW/pg2‚4.352848997641106e-05ÒÔò‘Ìû Æ _runtime‚6688.021469558 _step‚686 & lr-AdamW/pg1‚4.352848997641106e-05 & lr-AdamW/pg2‚4.352848997641106e-05 trainer/global_step‚8399 % _timestamp‚1.7884231882839286e+09®õ'Õ2ü - _wandbruntime‚6688  _runtime‚6688tÑÖÆç'ý áO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121826666 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484282880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111137996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7023484282880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7024140645376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367140 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494328330340864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539572841338368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158342928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7368986324992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481549053108264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169592 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7369156240384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884231882846422e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844ŽŸºaþ æO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158342928 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111137544 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487304189695488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚487304171692544 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121826666 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474551438218240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7023484282880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158362966 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111137996 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7023484282880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7024140645376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 280189632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487304171692544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6998286367140 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494328330340864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚539572841338368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158342928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494327655975424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7368986324992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_b –ŸF‹ atch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚481549053108264 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121826664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71075412 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532203878953472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 280169592 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 158362966 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494327655975424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474551421178880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7369156240384 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884231882846422e+09 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111150650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487360524731392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532265529691136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539635544738304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7370015047168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481605445151112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158381446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111150195 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6688.751483053 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303710720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40066615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960086528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487360542734336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381446 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303710720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7369855630848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40067068 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605ùfÑùõ&!ïM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532265505751040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7369855630848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494385502820864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40067068 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6999102802312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280222128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121840680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111150650 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280202062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487360524731392 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚532265529691136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280202062 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539635544738304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚481604773673788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474606342348800 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7370015047168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚481605445151112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487360542734336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158381446 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487360524731392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111150195 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6688.751483053 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7024303710720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40066615 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7024960086528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚453 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158361382 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚8399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487360542734336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6998448364348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158381446 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121840682 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121840680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71083582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280222128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494385502820864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494384828442112 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121840682 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7024303710720°Áƒž¡2‚!- _wandbruntime‚6688  _runtime‚6688;ð’\ƒ!Ö 2loss/SRqqt3/assignment_loss‚1.3619605827432892 0loss/FBt1/assignment_loss‚0.4271789414977479 1loss/FBt3/assignment_loss‚0.08215035206944776 /loss/FRt2/detection_loss‚3.0367184260795854 0loss/FRt1/assignment_loss‚2.4445343199703427 0loss/FRt4/assignment_loss‚1.3218662028617048 2loss/SRqqt1/assignment_loss‚1.4143988534521599 1loss/SRqqt3/detection_loss‚1.4508634470403194 /loss/FRt4/detection_loss‚1.7598460264345432 1loss/SRqqt1/detection_loss‚1.9335284309509473 0loss/SRqqt2/detection_loss‚2.700356488597804 0loss/FBt4/detection_loss‚0.44912515880390147 _step‚689 0loss/FRt2/assignment_loss‚3.7184424277009636 /loss/FBt3/detection_loss‚2.7131983836491904 %loss/total_loss‚41.77509486011144 1loss/FBt2/assignment_loss‚0.33565189491480396 0loss/FBt4/assignment_loss‚0.2295710833409801 ,loss/FRt1/detection_loss‚2.6667191516608 /loss/FBt1/detection_loss‚2.3741344023834574 /loss/FBt2/detection_loss‚2.7817644625902176 2loss/SRqqt4/assignment_loss‚0.6777668703531086 trainer/global_step‚8399 0loss/FRt3/assignment_loss‚2.6990108055373034 .loss/FRt3/detection_loss‚2.311203010380268 1loss/SRqqt4/detection_loss‚0.9455515733906921 epoch‚6 % _timestamp‚1.7884231890153012e+09 _runtime‚ 6688.75276375 2loss/SRqqt2/assignment_loss‚1.9395535637078614ÑUa„!Û 0loss/FRt3/assignment_loss‚2.6990108055373034 .loss/FRt3/detection_loss‚2.311203010380268 1loss/SRqqt4/detection_loss‚0.9455515733906921 epoch‚6 % _timestamp‚1.7884231890153012e+09 _runtime‚ 6688.75276375 2loss/SRqqt2/assignment_loss‚1.9395535637078614 2loss/SRqqt3/assignment_loss‚1.3619605827432892 0loss/FBt1/assignment_loss‚0.4271789414977479 1loss/FBt3/assignment_loss‚0.08215035206944776 /loss/FRt2/detection_loss‚3.0367184260795854 0loss/FRt1/assignment_loss‚2.4445343199703427 0loss/FRt4/assignment_loss‚1.3218662028617048 2loss/SRqqt1/assignment_loss‚1.4143988534521599 1loss/SRqqt3/detection_loss‚1.4508634470403194 /loss/FRt4/detection_loss‚1.7598460264345432 1loss/SRqqt1/detection_loss‚1.9335284309509473 0loss/SRqqt2/detection_loss‚2.700356488597804 0loss/FBt4/detection_loss‚0.44912515880390147 _step‚689 0loss/FRt2/assignment_loss‚3.7184424277009636 /loss/FBt3/detection_loss‚2.7131983836491904 %loss/total_loss‚41.77509486011144 1loss/FBt2/assignment_loss‚0.33565189491480396 0loss/FBt4/assignment_loss‚0.2295710833409801 ,loss/FRt1/detection_loss‚2.6667191516608 /loss/FBt1/detection_loss‚2.3741344023834574 /loss/FBt2/detection_loss‚2.7817644625902176 2loss/SRqqt4/assignment_loss‚0.6777668703531086 trainer/global_step‚8399±u•’…!‚(:‡ –ØäÔ¼Àœ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.511' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7022 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w€p†!‚(:å –ØäÔÿ’¨€ disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚4.671875" disk.dm-0.out‚27671.2568359375 disk.drbd2266.in‚0 network.recv‚ 2.4398e+06 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375* disk./.usagePercent‚17.419422857434032$ disk./.usageGB‚259.5619888305664 disk.nvme0c0n1.in‚4.671875' disk.nvme0c0n1.out‚27671.2568359375 disk.sda3.out‚ 383.296875 disk.rbd1.out‚ 3.373046875! proc.memory.rssMB‚ 2693.203125 network.sent‚ 1.9843232e+07% disk.nvme0n1.out‚27671.2568359375 cpu‚25.320798073040958 disk.drbd2265.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0* proc.memory.percent‚16.438007354736328 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0$ memory_percent‚90.81244468688965( proc.memory.availableMB‚ 1505.2890625 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.out‚ 383.296875 disk.sda.out‚ 383.296875 disk.nvme0n1.in‚4.671875¼o •’‡!‚(:‡ ¥ØäÔ¡ã~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.014' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.0028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕU< mˆ!‚(:â ¥ØäÔ†³€ network.recv‚ 2.446048e+06 disk.drbd2080.in‚0$ disk.nvme0c0n1.out‚ 27712.9296875$ disk./.usageGB‚259.5308074951172 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 27712.9296875 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0* proc.memory.percent‚16.438007354736328 disk.drbd2080.out‚0 disk.dm-1.out‚ 383.6640625 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.373046875 disk.sda.out‚ 383.6640625 disk.nvme0n1.in‚4.671875 cpu‚25.320193953936915 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1505.29296875* disk./.usagePercent‚17.417330251849155 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375$ memory_percent‚90.81242084503174 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.dm-0.out‚ 27712.9296875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda3.out‚ 383.6640625! proc.memory.rssMB‚ 2693.203125 network.sent‚ 1.9951689e+07 disk.sda1.in‚0 disk.sda2.out‚0#Íñ‰!‚(:’ ´ØäÔ⦗~ gpu.0.gpu‚41.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚136.892' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.378399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š½àódŠ!‚(:Ù ´ØäÔמ¶€ disk.drbd1716.in‚0 disk.dm-1.in‚ 126.77734375 network.sent‚ 1.9957984e+07 disk.nvme0n1.in‚4.671875! disk.nvme0c0n1.out‚ 27749.1875 disk.sda3.out‚ 384.3515625 disk.dm-0.in‚4.671875 disk.sda.out‚ 384.3515625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.438007354736328 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 384.3515625$ disk./.usageGB‚259.5578269958496 disk.drbd2265.in‚0 cpu‚25.319592959744313 disk.rbd1.out‚ 3.373046875* disk./.usagePercent‚17.419143553214177 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.out‚ 27749.1875 network.recv‚ 2.450798e+06 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0$ memory_percent‚90.81242084503174) proc.memory.availableMB‚ 1505.29296875 disk.nvme0n1.out‚ 27749.1875 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0! proc.memory.rssMB‚ 2693.203125ÚIR\2‹!- _wandbruntime‚6726  _runtime‚6726Í”Ø/ÇŒ!Á _step‚690 & lr-AdamW/pg1‚4.246724569864926e-05 & lr-AdamW/pg2‚4.246724569864926e-05 trainer/global_step‚8449 % _timestamp‚1.7884232271863103e+09 _runtime‚6726.923932055q<bÌ!Æ _runtime‚6726.923932055 _step‚690 & lr-AdamW/pg1‚4.246724569864926e-05 & lr-AdamW/pg2‚4.246724569864926e-05 trainer/global_step‚8449 % _timestamp‚1.7884232271863103e+09²FÐùƒ2Ž!- _wandbruntime‚6726  _runtime‚6726ñ—jjø!ßO UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542698567856128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111775913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542698738642944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293723648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7064470088704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7412322806272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484335094975672 trainer/global_step‚8449 _step‚691]1£Pê'!äO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 6726.92506891 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚497186968089088 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 146846720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122527464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71483814 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71483812 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚484335766452788 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281796444 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7412322806272 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159268980 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497186293723648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚484335094975672 trainer/global_step‚8449 _step‚691 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122527466 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚542698567856128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111775913 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚542698738642944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497186293723648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚490121841637888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7064470088704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6727.638720601 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484390815541196 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 trainer/global_step‚8449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497244140569088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71491980 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚490178194676736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7065945892352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚542761267658752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289516544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111788599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40296617 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497244140569088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7413045862400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542761088496640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497243466190336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945892352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40296175 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484391487018520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7039938141720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788423227900773e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111788155 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71491982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7413201084416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324»K¯ä“!ïM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6727.638720601 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159307498 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159287434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚484390815541196 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚477351548876800 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122541482 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159287434 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚490178176673792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281828914 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚477351531837440 trainer/global_step‚8449 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122541480 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122541480 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚490178176673792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497244140569088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71491980 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚490178194676736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7065945892352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚540 ä2761267658752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122541482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7065289516544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111788599 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40296617 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚497244140569088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7413045862400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚542761088496640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535348066574336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497243466190336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7065945892352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40296175 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7065289516544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚484391487018520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7039938141720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 $ _timestamp‚1.788423227900773e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281848980 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281848980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281828914 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159307498 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111788155 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71491982 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535348042634240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7413201084416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚486862574589936 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40583805 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚538005413165568 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚499809450167808 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112500665 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚545447767087616 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚499809450167808 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545447906371584 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚694 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809961000960 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6786.786549767 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 123253184 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7094012577792 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚486863082543688 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚8462 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7068385754696 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚492715437590016 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7094012577792 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 159904707 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚492715437590016 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 283141912 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 112500824 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚492715455592960 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚486862574589936 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7067894840304 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚40583805 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚538005413165568 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 159904707 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚499809450167808 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 123253182 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 112500665 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚545447767087616 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚71916860 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 159888730 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.´ ¼Pall.freed‚499809450167808 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚545447906371584 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚479794679749632 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚694 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚499809961000960 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚499809961000960 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6786.786549767 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 123253184 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 159888730 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139283968 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 7094012577792 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚486863082543688 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚492715455592960 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 trainer/global_step‚8462 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 7094505408000 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 283157891 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 7068385754696 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 283141912 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0¶=^Z±2¢!- _wandbruntime‚6787  _runtime‚6787C¬˜X)£!ÒR @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 283160020 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 112512052 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚492752267618816 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚486898231958832 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚538043312839168 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚545486147231744 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚71923366 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚486898739912584 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912009 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚499847156156416 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚159 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚499846645323264 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492752285621760 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7442834392576 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7068258449712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7094870534656 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7094377704448 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545486007947776 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚8462 % _timestamp‚1.7884232880068939e+09 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40588529 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492752267618816 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 159896032 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚499846645323264 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870534656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 123263990 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚538043288899072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 123263988 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377704448 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _runtime‚6787.744790718 _step‚695 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 123263988 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112511893 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 159896032 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚71923364 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚538043312839168 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚545486147231744 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚71923366 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚486898739912584 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 159912009 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚499847156156416 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚159 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚499846645323264 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚492752285621760 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7442834392576 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7068258449712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 7094870534656 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7094377704448 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚545486007947776 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚8462 % _timestamp‚1.7884232880068939e+09 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚40588529 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚492752267618816 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 159896032 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚499846645323264 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7094870534656 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 123263990 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚538043288899072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 123263988 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 9955586048 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139283968 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚479829990548480 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7094377704448 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 _runtime‚6787.744790718 _step‚695 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 123263988 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 112511893 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944· F6’¥!‚(:‡ ÿØäÔ˜ý~ gpu.0.gpu‚52.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ÆXtl¦!‚(:á ÿØäÔŸ²« disk.rbd1.out‚ 3.373046875 disk.sda3.in‚ 126.77734375 cpu‚25.358476904137586 disk.sda2.in‚0 disk.drbd2080.in‚0$ memory_percent‚91.29667282104492 proc.cpu.threads‚17 disk.sda.out‚ 387.0546875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-1.out‚ 387.0546875 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 27966.6953125 disk.rbd0.in‚0 disk.sda3.out‚ 387.0546875 network.sent‚ 2.0088656e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2772.296875* disk./.usagePercent‚17.421430980532612% disk./.usageGB‚259.59191131591797 disk.dm-0.out‚ 27966.6953125 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875$ disk.nvme0c0n1.out‚ 27966.6953125' proc.memory.availableMB‚ 1425.953125* proc.memory.percent‚16.920757293701172 network.recv‚ 2.471938e+06 disk.dm-0.in‚4.671875’(<+2§!- _wandbruntime‚6800  _runtime‚6800§Œ™Ça¨!Ȧ 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08711723238229752 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864809513092041 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4294871687889099 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20086683332920074 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2526845932006836 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1339762955904007 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5745306611061096 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18741221725940704 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902381896972656 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921019077301025 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9770832657814026 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35225751996040344 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39701637625694275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5892243385314941 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25742170214653015 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45053789019584656 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5901243090629578 +jet/accuracy_2_of_2‚0.10989528894424438 *jet/accuracy_2_of_3‚0.2461629956960678 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08888889104127884 +jet/accuracy_4_of_6‚0.07692307978868484 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2645213007926941 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5475510358810425 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5496202111244202 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5583875775337219 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9867283701896667 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.182759091258049 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3840278089046478 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6066427826881409 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02040705643594265 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12442529946565628 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32574355602264404 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3h•î|FRt1SRqqt0FBt/SRqqt_purity‚0.5666666626930237 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03578723594546318 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.501168429851532 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5385003685951233 *jet/accuracy_4_of_5‚0.0098566310480237 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8842456340789795 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_1_of_2‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975618720054626 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5340240001678467 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.574840247631073 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0912153571844101 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011111111380159855 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.03703703731298447 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13469022512435913 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03739592060446739 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5906032919883728 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5909178256988525 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5013882517814636 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10998048633337021 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5300276279449463 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4859195053577423 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20776332914829254 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25550299882888794 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15990746021270752 (particle/accuracy‚0.8474031686782837 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912829399108887 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3589635193347931 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40417519211769104 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5701237916946411 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.331845223903656 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16936936974525452 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_7_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5063218474388123 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697669982910156 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24041374027729034 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_7‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4444443881511688 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11505749076604843 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.511963963508606 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5611956119537354 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5096490979194641 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18895122408866882 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4764706492424011 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2937358021736145 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9769841432571411 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5013567805290222 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9884209036827087 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 *jet/accuracy_1_of_1‚0.5084683299064636 7validation_average_jet_accuracy‚0.45100346207618713 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1476716548204422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19079430401325226 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802686214447 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19936779141426086 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15792348980903625 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015698907896876335 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12729431688785553 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5585106015205383 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 -jet/accuracy_4_of_4‚0.0021919880528002977 % _timestamp‚1.7884233007089014e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 _runtime‚6800.448988647 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35076799988746643 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07136393338441849 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5951388478279114 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6380108594894409 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5690168142318726 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5585106015205383 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5627965331077576 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20061726868152618 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5854386687278748 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5506200790405273 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.022821662947535515 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13224051892757416 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15985362231731415 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13060106337070465 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.969468355178833 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3965773582458496 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1101873442530632 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18065959215164185 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5367856025695801 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08723316341638565 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13101784884929657 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6293103694915771 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900357723236084 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835630655288696 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887241721153259 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6337007880210876 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12242621183395386 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8188612461090088 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0022675737272948027 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140220165253 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5719610452651978 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28564226627349854 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007777859922498465 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5367063879966736 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3027874529361725 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 +jet/accuracy_2_of_4‚0.37152910232543945 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47973865270614624 epoch‚6 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.459509015083313 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5071491599082947 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990832507610321 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5899015069007874 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 +particle/specificity‚0.9481933116912842 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6029863357543945 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36617422103881836 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5776785612106323 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04210091754794121 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10321637243032455 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3453703224658966 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4513888955116272 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09779016673564911 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5560610890388489 +jet/accuracy_3_of_6‚0.17307692766189575 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.49215662479400635 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5418180227279663 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/f_score‚0.38247302174568176 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08155494183301926 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6080247163772583 *jet/accuracy_1_of_4‚0.9232808947563171 particle/accuracy_3_of_5‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5112589001655579 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07405120134353638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9491615295410156 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 _step‚696 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979956746101379 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5345907807350159 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5956959128379822 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13813190162181854 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15977968275547028 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23593245446681976 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13813132047653198 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 )loss/val_total_loss‚43.05254916652466 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2849220931529999 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9826869368553162 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31813356280326843 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2614942193031311 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚8462 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.039147138595581055 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06995464861392975 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5995370745658875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5062068104743958 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40030866861343384 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5921083092689514 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08898808807134628 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06130268797278404 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5278314352035522 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5825049877166748 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4705386757850647 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5834986567497253 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07741838693618774 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889453649520874 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821894764900208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_3_of_3‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18264704942703247 *jet/accuracy_2_of_5‚0.4795442819595337 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166269302368 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40758875012397766 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3779650330543518 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15962551534175873 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +jet/accuracy_3_of_4‚0.06689666956663132 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4444444477558136 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5516923069953918 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5595661997795105 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1276489496231079 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 particle/accuracy_5_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4522038400173187 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04125063866376877 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24847547709941864 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15977968275547028 jet/accuracy_1_of_6‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09964033216238022 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09682600945234299 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01972177065908909 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4287281036376953 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5365731120109558 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6896400451660156 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2708333432674408 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4976640045642853 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12848743796348572 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09630095958709717 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5867817997932434 *jet/accuracy_1_of_2‚0.7787728309631348 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1ÝôïL©!ͦ :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4705386757850647 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5834986567497253 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("PuritÝ Èùy/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_6‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07741838693618774 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9889453649520874 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9821894764900208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_3_of_3‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18264704942703247 *jet/accuracy_2_of_5‚0.4795442819595337 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166269302368 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40758875012397766 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3779650330543518 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15962551534175873 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 +jet/accuracy_3_of_4‚0.06689666956663132 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4444444477558136 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5516923069953918 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5595661997795105 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1276489496231079 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.008064515888690948 particle/accuracy_5_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4522038400173187 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04125063866376877 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24847547709941864 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15977968275547028 jet/accuracy_1_of_6‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09964033216238022 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09682600945234299 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01972177065908909 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4287281036376953 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5365731120109558 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0019919080659747124 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0.5 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_6‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6896400451660156 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2708333432674408 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4976640045642853 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12848743796348572 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09630095958709717 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5867817997932434 *jet/accuracy_1_of_2‚0.7787728309631348 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6458333134651184 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08711723238229752 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9864809513092041 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.4294871687889099 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.262037068605423 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20086683332920074 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2526845932006836 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1339762955904007 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5745306611061096 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18741221725940704 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9902381896972656 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921019077301025 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9770832657814026 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35225751996040344 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39701637625694275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5892243385314941 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25742170214653015 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45053789019584656 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5901243090629578 +jet/accuracy_2_of_2‚0.10989528894424438 *jet/accuracy_2_of_3‚0.2461629956960678 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08888889104127884 +jet/accuracy_4_of_6‚0.07692307978868484 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2645213007926941 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5475510358810425 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5496202111244202 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5583875775337219 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9867283701896667 8"Purity/1FRt0SRqqt*FBt/event_purity‚0.182759091258049 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3840278089046478 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.40625 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6066427826881409 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 particle/accuracy_1_of_1‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02040705643594265 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12442529946565628 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.32574355602264404 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5666666626930237 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03578723594546318 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.501168429851532 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5385003685951233 *jet/accuracy_4_of_5‚0.0098566310480237 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8842456340789795 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_1_of_2‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975618720054626 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5340240001678467 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.574840247631073 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0912153571844101 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.7017543911933899 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011111111380159855 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.03703703731298447 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13469022512435913 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03739592060446739 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5906032919883728 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611342906952 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5909178256988525 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5013882517814636 ;"Purity/1FRt2SRqqt0FBt/event_purity‚0.002765753073617816 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10998048633337021 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5300276279449463 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4859195053577423 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20776332914829254 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.25550299882888794 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15990746021270752 (particle/accuracy‚0.8474031686782837 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9912829399108887 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3589635193347931 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40417519211769104 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5701237916946411 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.331845223903656 particle/accuracy_1_of_3‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16936936974525452 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 particle/accuracy_4_of_7‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_7_of_7‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5063218474388123 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.40697669982910156 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24041374027729034 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_6_of_7‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121212482452393 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4444443881511688 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6458333134651184 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11505749076604843 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.511963963508606 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5611956119537354 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5096490979194641 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18895122408866882 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4764706492424011 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2937358021736145 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9769841432571411 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5013567805290222 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9884209036827087 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 *jet/accuracy_1_of_1‚0.5084683299064636 7validation_average_jet_accuracy‚0.45100346207618713 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1476716548204422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19079430401325226 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4135802686214447 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19936779141426086 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.15792348980903625 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015698907896876335 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12729431688785553 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 particle/accuracy_5_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5585106015205383 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 -jet/accuracy_4_of_4‚0.0021919880528002977 % _timestamp‚1.7884233007089014e+09 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 _runtime‚6800.448988647 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35076799988746643 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07136393338441849 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5951388478279114 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6380108594894409 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5690168142318726 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5585106015205383 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5627965331077576 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20061726868152618 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5854386687278748 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5506200790405273 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_3‚0.022821662947535515 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13224051892757416 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15985362231731415 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13060106337070465 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.969468355178833 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3965773582458496 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1101873442530632 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18065959215164185 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5367856025695801 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08723316341638565 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13101784884929657 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.6293103694915771 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900357723236084 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835630655288696 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887241721153259 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6337007880210876 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.14393939077854156 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt1SRqqt3FBt/event_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12242621183395386 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8188612461090088 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0022675737272948027 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1041666641831398 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07456140220165253 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5719610452651978 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28564226627349854 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007777859922498465 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5367063879966736 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 jet/accuracy_4_of_7‚0 jet/accuracy_7_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_6‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3027874529361725 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 +jet/accuracy_2_of_4‚0.37152910232543945 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.47973865270614624 epoch‚6 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.459509015083313 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5071491599082947 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.990832507610321 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5899015069007874 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.37878790497779846 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 +particle/specificity‚0.9481933116912842 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.6029863357543945 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36617422103881836 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 *"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5776785612106323 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4814814329147339 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04210091754794121 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10321637243032455 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3453703224658966 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4513888955116272 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09779016673564911 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5560610890388489 +jet/accuracy_3_of_6‚0.17307692766189575 jet/accuracy_6_of_7‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.49215662479400635 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5418180227279663 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/f_score‚0.38247302174568176 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08155494183301926 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.6080247163772583 *jet/accuracy_1_of_4‚0.9232808947563171 particle/accuracy_3_of_5‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5112589001655579 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07405120134353638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9491615295410156 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 _step‚696 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9979956746101379 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5345907807350159 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5956959128379822 ("Purity/1FRt1SRqqt2FBZ¨ÄNWt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1230158805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13813190162181854 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15977968275547028 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23593245446681976 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13813132047653198 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 )loss/val_total_loss‚43.05254916652466 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2849220931529999 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9826869368553162 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31813356280326843 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2614942193031311 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 trainer/global_step‚8462 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.039147138595581055 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06995464861392975 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5995370745658875 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5062068104743958 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.40030866861343384 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5921083092689514 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08898808807134628 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.06130268797278404 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_2_of_7‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5278314352035522 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5825049877166748 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043¸y«Ùu’ª!‚(:‡ ŽÙäÔú€½~ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚172.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.4882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ô¼¢v«!‚(:ë ŽÙäԟ׫ disk.drbd1716.out‚0 disk.sda3.out‚ 388.0859375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 2.0197287e+07* disk./.usagePercent‚17.423979279528442 disk.dm-0.in‚4.671875 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚28514.95556640625$ memory_percent‚90.24007320404053 disk.dm-1.out‚ 388.0859375( disk.nvme0c0n1.out‚28514.95556640625 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0* proc.memory.percent‚16.920995712280273 disk.drbd2265.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚28514.95556640625 disk.sda.out‚ 388.0859375" proc.memory.rssMB‚ 2772.3359375 disk.dm-1.in‚ 126.77734375) proc.memory.availableMB‚ 1599.06640625 cpu‚25.36138391105308 proc.cpu.threads‚17 network.recv‚ 2.478662e+06! disk./.usageGB‚259.6298828125 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.458984375 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0(„Äù¬!‚(:… ÙäÔ˹~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I.|­!‚(:ñ ÙäÔ‚£€ disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚28612.48583984375 disk.sda3.in‚ 126.77734375 network.sent‚ 2.0321207e+07 network.recv‚ 2.484839e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚4.671875" proc.memory.rssMB‚ 2772.3359375* proc.memory.percent‚16.920995712280273 cpu‚25.360710387431876 disk.rbd1.out‚ 3.521484375 disk.sda.out‚ 388.58984375 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 388.58984375 disk.sda.in‚ 126.77734375 disk.drbd2265.in‚0 disk.sda2.in‚0# disk.dm-0.out‚28612.48583984375 disk.drbd1716.out‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1598.81640625 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚28612.48583984375 disk.sda3.out‚ 388.58984375) disk./.usagePercent‚17.42326860261339 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0$ memory_percent‚90.24159908294678 proc.cpu.threads‚17$ disk./.usageGB‚259.6192932128906o°ÐG2®!- _wandbruntime‚6831  _runtime‚6831ó¹Ö¢Ç¯!Á & lr-AdamW/pg1‚4.141472832854511e-05 & lr-AdamW/pg2‚4.141472832854511e-05 trainer/global_step‚8499 % _timestamp‚1.7884233321914701e+09 _runtime‚6831.929023235 _step‚697Ù(»áÌ°!Æ & lr-AdamW/pg1‚4.141472832854511e-05 & lr-AdamW/pg2‚4.141472832854511e-05 trainer/global_step‚8499 % _timestamp‚1.7884233321914701e+09 _runtime‚6831.929023235 _step‚697¹÷Àâd2±!- _wandbruntime‚6831  _runtime‚6831Kx'ç'²!áO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 trainer/global_step‚8499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540755123930624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502388917621760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884233321921964e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113112584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40811065 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128475864576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6831.930067356 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7478425676800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495259767391744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548233525667328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548233711134208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7478587203584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113113031 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908›Ö¢ì'³!æO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚489368239668022 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚502388243256320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123909042 trainer/global_step‚8499 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚540755123930624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚502388917621760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160677033 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚502388917621760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482265358121984 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123909042 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚495259785394688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160677033 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72301964 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884233321921964e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113112584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40811065 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7128475864576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚6831.930067356 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123909044 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7478425676800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚482265341082624 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚495259767391744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚495259767391744 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚495259785394688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284586077 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚548233525667328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚548233711134208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7478587203584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 113113031 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160656995 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72301966 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Jµ!ëM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40815078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951668224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 % _timestamp‚1.7884233329014006e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6832.639444277 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548296223237632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495316138433536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295292416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚8499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675449 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113125210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113125658 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7479287055872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7479448569344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0äéö&¶!ðM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7129295292416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚8499 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160675449 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7129951668224 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160695513 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚502446090101760 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113125210 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113125658 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7479287055872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123923058 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72310132 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160695513 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7479448569344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482320262252544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚502445415723008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7103697981210 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284598507 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489423960233754 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40815078 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7129951668224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 % _timestamp‚1.7884233329014006e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚495316138433536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284618573 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284598507 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123923060 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚495316120430592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7129295292416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _runtime‚6832.639444277 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548296223237632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123923060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7103043543246 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚495316138433536 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160675449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872»o2pÇ2·!- _wandbruntime‚6832  _runtime‚6832\±„sU¸!Ï 1loss/FBt1/assignment_loss‚0.16809653128600782 1loss/SRqqt4/detection_loss‚1.3727196531577244 .loss/FBt3/detection_loss‚3.600163141886393 2loss/SRqqt4/assignment_loss‚0.9640739471985769 ,loss/FRt3/detection_loss‚2.6658047972755 % _timestamp‚1.7884233329035907e+09 .loss/FRt2/assignment_loss‚2.78420382585281 2loss/SRqqt2/assignment_loss‚1.8413625370491953 0loss/SRqqt3/detection_loss‚1.796908549964428 _runtime‚6832.641079583 0loss/FRt1/assignment_loss‚2.4484333360896393 0loss/FRt4/assignment_loss‚1.4040384856717927 2loss/SRqqt1/assignment_loss‚1.3200337167262384 2loss/SRqqt3/assignment_loss‚1.2218629064521305 0loss/FBt2/assignment_loss‚1.0998199419584125 1loss/SRqqt1/detection_loss‚1.9041265169887438 &loss/total_loss‚43.985237052951106 /loss/FBt2/detection_loss‚2.4868497371673586 .loss/FRt1/detection_loss‚3.024008061955957 0loss/SRqqt2/detection_loss‚2.524608212358811 /loss/FBt1/detection_loss‚2.9171079927020602 epoch‚7 trainer/global_step‚8499 /loss/FRt3/assignment_loss‚2.905361073938283 0loss/FBt3/assignment_loss‚0.4711110470816493 0loss/FBt4/assignment_loss‚0.1791239115982949 .loss/FRt2/detection_loss‚2.749332615962395 /loss/FBt4/detection_loss‚0.4398888167981283 _step‚700 .loss/FRt4/detection_loss‚1.696197695830571]üD¬Z¹!Ô trainer/global_step‚8499 /loss/FRt3/assignment_loss‚2.905361073938283 0loss/FBt3/assignment_loss‚0.4711110470816493 0loss/FBt4/assignment_loss‚0.1791239115982949 .loss/FRt2/detection_loss‚2.749332615962395 /loss/FBt4/detection_loss‚0.4398888167981283 _step‚700 .loss/FRt4/detection_loss‚1.696197695830571 1loss/FBt1/assignment_loss‚0.16809653128600782 1loss/SRqqt4/detection_loss‚1.3727196531577244 .loss/FBt3/detection_loss‚3.600163141886393 2loss/SRqqt4/assignment_loss‚0.9640739471985769 ,loss/FRt3/detection_loss‚2.6658047972755 % _timestamp‚1.7884233329035907e+09 .loss/FRt2/assignment_loss‚2.78420382585281 2loss/SRqqt2/assignment_loss‚1.8413625370491953 0loss/SRqqt3/detection_loss‚1.796908549964428 _runtime‚6832.641079583 0loss/FRt1/assignment_loss‚2.4484333360896393 0loss/FRt4/assignment_loss‚1.4040384856717927 2loss/SRqqt1/assignment_loss‚1.3200337167262384 2loss/SRqqt3/assignment_loss‚1.2218629064521305 0loss/FBt2/assignment_loss‚1.0998199419584125 1loss/SRqqt1/detection_loss‚1.9041265169887438 &loss/total_loss‚43.985237052951106 /loss/FBt2/detection_loss‚2.4868497371673586 .loss/FRt1/detection_loss‚3.024008061955957 0loss/SRqqt2/detection_loss‚2.524608212358811 /loss/FBt1/detection_loss‚2.9171079927020602 epoch‚7¼ Z’º!‚(:‡ ¬ÙäÔßì¢~ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.492' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{†Ä‚v»!‚(:ë ¬ÙäÔÇÖ†) disk./.usagePercent‚17.42473270970171 disk.sda1.in‚0& disk.nvme0n1.out‚29088.76318359375 disk.drbd2080.out‚0 cpu‚25.36006202429619 network.recv‚ 2.489557e+06% disk./.usageGB‚259.64110946655273 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚29088.76318359375 disk.sda2.in‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.920995712280273 disk.rbd0.in‚0# disk.dm-0.out‚29088.76318359375& proc.memory.availableMB‚ 1598.78125 network.sent‚ 2.0327571e+07 disk.rbd1.in‚0 disk.dm-0.in‚4.671875" proc.memory.rssMB‚ 2772.3359375 disk.rbd1.out‚ 3.521484375 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚90.24181365966797 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 389.1953125 disk.dm-1.out‚ 389.1953125 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 389.1953125 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0êÛ½{¼!‚(:… »ÙäÔ£§”~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.09' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.418 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h¨•u½!‚(:ê »ÙäÔæ—ü€ disk.drbd1716.in‚0& disk.nvme0n1.out‚29144.48681640625 disk.drbd2080.in‚0 disk.dm-1.out‚ 389.640625) proc.memory.availableMB‚ 1598.78515625 disk.sda.out‚ 389.640625 disk.nvme0c0n1.in‚4.671875 disk.sda2.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚29144.48681640625 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2772.3359375) disk./.usagePercent‚17.42550662047772 disk.rbd1.in‚0( disk.nvme0c0n1.out‚29144.48681640625 disk.drbd2266.out‚0 cpu‚25.3593503977956$ disk./.usageGB‚259.6526412963867 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ memory_percent‚90.24178981781006* proc.memory.percent‚16.920995712280273 network.recv‚ 2.494842e+06 disk.rbd1.out‚ 3.5537109375 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 389.640625 disk.dm-0.in‚4.671875 disk.drbd1716.out‚0 disk.dm-1.in‚ 126.77734375 network.sent‚ 2.0435424e+07 disk.sda1.in‚0òA;¼2¾!- _wandbruntime‚6869  _runtime‚6869’‹GÇ¿!Á & lr-AdamW/pg1‚4.037115720914096e-05 & lr-AdamW/pg2‚4.037115720914096e-05 trainer/global_step‚8549 % _timestamp‚1.7884233694838223e+09 _runtime‚6869.221632097 _step‚701F÷ÁÌÀ!Æ & lr-AdamW/pg1‚4.037115720914096e-05 & lr-AdamW/pg2‚4.037115720914096e-05 trainer/global_step‚8549 % _timestamp‚1.7884233694838223e+09 _runtime‚6869.221632097 _step‚701½æôkz2Á!- _wandbruntime‚6869  _runtime‚6869üŸœ=æ'Â!àO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551359126716416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124609842 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118032896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41036245 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7521659758080 _runtime‚ 6869.22272536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551359320571904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7170118032896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113746169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430¼G„#Ã!åO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505247555369984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 113746169 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7169461670400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505247555369984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551359126716416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498077437337088 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124609842 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7170118032896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41036245 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505246881004544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124609842 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498077419334144 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286192889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚485010564649984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7521659758080 _runtime‚ 6869.22272536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551359320571904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7170118032896 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430¾zmÀ¥2Ä!- _wandbruntime‚6869  _runtime‚6869£èš.ñ&Å!ëM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113758751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7522520908800 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚492210002101162 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6869.916821519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551421653298176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727849984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41040642 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551421832460288 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7522365686784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053471232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884233701788037e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚703 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621565 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7170281098240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498133790375936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚8549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 2862454259 ÃÎö&Æ!ðM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41040642 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286225359 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551421832460288 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7522365686784 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7170937474048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚505304053471232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7170281098240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚492209330623838 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505304727849984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543899287611392 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 % _timestamp‚1.7884233701788037e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 _step‚703 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161621565 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498133790375936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7170281098240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124623860 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚485065451741184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124623858 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7170937474048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72718534 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚485065468780544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498133790375936 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7144533320618 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543899311551488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚8549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7143878882654 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498133772372992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286245425 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113758751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7522520908800 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124623860 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72718532 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚492210002101162 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 _runtime‚6869.916821519 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚423 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551421653298176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286245425 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505304727849984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498133772372992¿Yï2Ç!- _wandbruntime‚6869  _runtime‚6869ƒ2²RUÈ!Ï /loss/FRt1/detection_loss‚2.0221130081585477 trainer/global_step‚8549 2loss/SRqqt4/assignment_loss‚0.7602690006858748 0loss/FBt1/assignment_loss‚0.2549486519502742 1loss/FBt4/assignment_loss‚0.19363703101516644 % _timestamp‚1.7884233701805246e+09 1loss/FBt2/assignment_loss‚0.38185226917266846 .loss/FRt4/detection_loss‚1.761629818973166 0loss/FRt4/assignment_loss‚1.2652020597899403 /loss/FBt1/detection_loss‚3.3771479981286183 2loss/SRqqt2/assignment_loss‚1.8375271303722491 1loss/FBt3/assignment_loss‚0.41663633682765067 /loss/SRqqt3/detection_loss‚1.80764825469149 /loss/FBt3/detection_loss‚2.3598180413246155 _runtime‚6869.918000785 1loss/SRqqt1/detection_loss‚1.8551599813455886 -loss/FBt2/detection_loss‚3.04579838514328 /loss/FBt4/detection_loss‚0.5813942895454849 0loss/FRt1/assignment_loss‚3.0875828053270067 /loss/FRt3/detection_loss‚2.4652642905712128 &loss/total_loss‚42.823055587769346 epoch‚7 /loss/FRt2/assignment_loss‚4.151056352588865 /loss/FRt3/assignment_loss‚2.219947830932896 /loss/SRqqt1/assignment_loss‚0.9824732423506 .loss/FRt2/detection_loss‚2.871596192320188 /loss/SRqqt2/detection_loss‚2.82751432290444 1loss/SRqqt4/detection_loss‚0.9371378290725042 _step‚704 1loss/SRqqt3/assignment_loss‚1.359700464577015¨XìZÉ!Ô 2loss/SRqqt4/assignment_loss‚0.7602690006858748 0loss/FBt1/assignment_loss‚0.2549486519502742 1loss/FBt4/assignment_loss‚0.19363703101516644 % _timestamp‚1.7884233701805246e+09 1loss/FBt2/assignment_loss‚0.38185226917266846 .loss/FRt4/detection_loss‚1.761629818973166 0loss/FRt4/assignment_loss‚1.2652020597899403 /loss/FBt1/detection_loss‚3.3771479981286183 2loss/SRqqt2/assignment_loss‚1.8375271303722491 1loss/FBt3/assignment_loss‚0.41663633682765067 /loss/SRqqt3/detection_loss‚1.80764825469149 /loss/FBt3/detection_loss‚2.3598180413246155 _runtime‚6869.918000785 1loss/SRqqt1/detection_loss‚1.8551599813455886 -loss/FBt2/detection_loss‚3.04579838514328 /loss/FBt4/detection_loss‚0.5813942895454849 0loss/FRt1/assignment_loss‚3.0875828053270067 /loss/FRt3/detection_loss‚2.4652642905712128 &loss/total_loss‚42.823055587769346 epoch‚7 /loss/FRt2/assignment_loss‚4.151056352588865 /loss/FRt3/assignment_loss‚2.219947830932896 /loss/SRqqt1/assignment_loss‚0.9824732423506 .loss/FRt2/detection_loss‚2.871596192320188 /loss/SRqqt2/detection_loss‚2.82751432290444 1loss/SRqqt4/detection_loss‚0.9371378290725042 _step‚704 1loss/SRqqt3/assignment_loss‚1.359700464577015 /loss/FRt1/detection_loss‚2.0221130081585477 trainer/global_step‚8549ÀÈøïÊ!‚(:‡ ÊÙäÔ€…‘~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPvvÖŒercent‚31.3156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fí³ù~Ë!‚(:ó ÊÙäԜױ€ network.recv‚ 2.499164e+06# disk.dm-0.out‚29189.35791015625 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.nvme0n1.in‚4.671875 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3359375 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.sent‚ 2.0441647e+07 disk.drbd2265.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0* disk./.usagePercent‚17.426376022063998 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.5537109375 disk.sda2.in‚0 cpu‚25.358681941626564 disk.rbd0.out‚0$ memory_percent‚90.24345874786377$ disk./.usageGB‚259.6655960083008 disk.dm-1.out‚ 390.23046875 disk.rbd1.in‚0 disk.sda.out‚ 390.23046875 disk.sda3.out‚ 390.23046875) proc.memory.availableMB‚ 1598.51171875 disk.drbd2080.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚29189.35791015625( disk.nvme0c0n1.out‚29189.35791015625* proc.memory.percent‚16.920995712280273¤w9!’Ì!‚(:‡ ÙÙäÔÚ‡¥~ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ²@pÍ!‚(:å ÙÙäÔßø± disk.dm-0.in‚4.671875 cpu‚25.358004412405617 proc.cpu.threads‚17 disk.drbd1716.in‚0" disk.dm-0.out‚29245.8974609375 disk.dm-1.out‚390.6875 network.sent‚ 2.0550105e+07 network.recv‚ 2.505168e+06$ disk./.usageGB‚259.6771659851074 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875 disk.dm-1.in‚ 126.77734375$ memory_percent‚90.24217128753662" proc.memory.rssMB‚ 2772.3359375 disk.nvme0n1.in‚4.671875' disk.nvme0c0n1.out‚29245.8974609375 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.61669921875* proc.memory.percent‚16.920995712280273 disk.sda3.out‚390.6875* disk./.usagePercent‚17.427152492915347 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.sda.out‚390.6875% disk.nvme0n1.out‚29245.8974609375 disk.drbd2265.out‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1598.72265625myƒzœÎ!‚(:‘ èÙäÔ“ª®~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.531' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.50620000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ÏšSwÏ!‚(:ì èÙäÔæ˾ network.sent‚ 2.0556406e+07 disk.rbd1.out‚ 3.61669921875 disk.sda.out‚ 391.21484375" proc.memory.rssMB‚ 2772.3359375 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚259.6841354370117 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 391.21484375 cpu‚25.3573004919812 network.recv‚ 2.509738e+06 disk.sda1.out‚0" disk.dm-0.out‚29294.9130859375 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.427620218680406 disk.nvme0n1.in‚4.671875' disk.nvme0c0n1.out‚29294.9130859375 disk.drbd2266.in‚0$ memory_percent‚90.24219512939453& proc.memory.availableMB‚ 1598.71875* proc.memory.percent‚16.920995712280273 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚29294.9130859375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 391.21484375 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0Æ!72Ð!- _wandbruntime‚6906  _runtime‚6906¤£CtÇÑ!Á trainer/global_step‚8599 % _timestamp‚1.7884234064620106e+09 _runtime‚6906.199533044 _step‚705 & lr-AdamW/pg1‚3.933674981909396e-05 & lr-AdamW/pg2‚3.933674981909396e-05¤ –ˆÌÒ!Æ trainer/global_step‚8599 % _timestamp‚1.7884234064620106e+09 _runtime‚6906.199533044 _step‚705 & lr-AdamW/pg1‚3.933674981909396e-05 & lr-AdamW/pg2‚3.933674981909396e-05Áû±¸}2Ó!- _wandbruntime‚6906  _runtime‚6906Ùæ'Ô!àO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7211103838720 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚546920173756928 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487755754138624 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162529137 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚6906.200629313 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7564514938368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508105518752768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788423406462815e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193118208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554484516682752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162509099 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41261083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41260634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529137 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925722 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103838720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554484712635392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125310642 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7184552224854 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚500895089279488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7210447476224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287839781 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚494940323402838 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162509099 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41261083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41260634 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487755771177984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162529137 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494939651925722 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7211103838720 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚554484712635392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7564514938368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚508105518752768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287819741 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚500895071276544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73118766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125310642 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788423406462815e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚508106193118208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚508105518752768 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚554484516682752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚500895071276544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430·#s°2Ö!- _wandbruntime‚6907  _runtime‚6907ƒ^¢´…×!êM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788423407415624e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554547210268672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6907.154036912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114392013 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554547012232192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73126932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41265528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114392462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7565187737600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeì>Rk viceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162527553 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7565361833984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447¿Bíõ&Ø!ïM FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 162547617 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7184714222062 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 trainer/global_step‚8599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287852211 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125324658 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162527553 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7565361833984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287872277 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287872277 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125324660 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚500951442318336 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487810658269184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 162527553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788423407415624e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚508162691219456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚500951424315392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚554547210268672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494995372491246 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚6907.154036912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114392013 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125324658 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494996043968570 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7185368660026 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125324660 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7211923279872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73126934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚554547012232192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73126932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41265528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114392462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚546981824494592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚546981848434688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7565187737600 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚500951442318336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ÃRþ 2Ù!- _wandbruntime‚6907  _runtime‚6907Y"Æ“XÚ!Ò /loss/FRt2/assignment_loss‚3.057317500934005 2loss/SRqqt3/assignment_loss‚1.3849039811129664 1loss/FBt3/assignment_loss‚0.08849307411001064 /loss/FRt3/detection_loss‚2.4383942333032502 0loss/SRqqt2/detection_loss‚2.729708202860572 trainer/global_step‚8599 _runtime‚6907.155080321 /loss/FRt4/detection_loss‚1.7453393809821294 /loss/FBt3/detection_loss‚3.1736648797988893 /loss/FBt4/detection_loss‚0.4543396701597992 % _timestamp‚1.7884234074175367e+09 %loss/total_loss‚44.69544673473424 .loss/FRt1/assignment_loss‚3.77298975313032 0loss/FRt4/assignment_loss‚1.4989434221516484 2loss/SRqqt4/assignment_loss‚0.8592018662149914 0loss/FBt1/assignment_loss‚0.3885472582653165 1loss/SRqqt1/detection_loss‚2.2430557519197465 1loss/SRqqt3/detection_loss‚1.9264758103618436 _step‚708 2loss/SRqqt1/assignment_loss‚1.3885430177673697 /loss/FRt2/detection_loss‚3.1043038703501225 .loss/FBt2/detection_loss‚2.220092540437525 0loss/FRt3/assignment_loss‚3.1041285181945226 /loss/FBt4/assignment_loss‚0.130098419537553 1loss/SRqqt4/detection_loss‚0.9405116459561719 epoch‚7 2loss/SRqqt2/assignment_loss‚1.8026108687574214 1loss/FBt2/assignment_loss‚0.09128510574704375 /loss/FRt1/detection_loss‚2.4367504393732227 ,loss/FBt1/detection_loss‚3.7157475233078¯ÉBC]Û!× 0loss/FRt4/assignment_loss‚1.4989434221516484 2loss/SRqqt4/assignment_loss‚0.8592018662149914 0loss/FBt1/assignment_loss‚0.3885472582653165 1loss/SRqqt1/detection_loss‚2.2430557519197465 1loss/SRqqt3/detection_loss‚1.9264758103618436 _step‚708 2loss/SRqqt1/assignment_loss‚1.3885430177673697 /loss/FRt2/detection_loss‚3.1043038703501225 .loss/FBt2/detection_loss‚2.220092540437525 0loss/FRt3/assignment_loss‚3.1041285181945226 /loss/FBt4/assignment_loss‚0.130098419537553 1loss/SRqqt4/detection_loss‚0.9405116459561719 epoch‚7 2loss/SRqqt2/assignment_loss‚1.8026108687574214 1loss/FBt2/assignment_loss‚0.09128510574704375 /loss/FRt1/detection_loss‚2.4367504393732227 ,loss/FBt1/detection_loss‚3.7157475233078 /loss/FRt2/assignment_loss‚3.057317500934005 2loss/SRqqt3/assignment_loss‚1.3849039811129664 1loss/FBt3/assignment_loss‚0.08849307411001064 /loss/FRt3/detection_loss‚2.4383942333032502 0loss/SRqqt2/detection_loss‚2.729708202860572 trainer/global_step‚8599 _runtime‚6907.155080321 /loss/FRt4/detection_loss‚1.7453393809821294 /loss/FBt3/detection_loss‚3.1736648797988893 /loss/FBt4/detection_loss‚0.4543396701597992 % _timestamp‚1.7884234074175367e+09 %loss/total_loss‚44.69544673473424 .loss/FRt1/assignment_loss‚3.77298975313032ÄÄž¬j’Ü!‚(:‡ ÷ÙäÔ±²~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'œ:´xÝ!‚(:í ÷ÙäÔßæЀ disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚4.671875# memory_percent‚90.2423620223999 cpu‚25.35667894656489 proc.cpu.threads‚17$ disk./.usageGB‚259.6902198791504 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375* proc.memory.percent‚16.920995712280273 disk.sda.out‚ 391.67578125 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 3.6796875& disk.nvme0n1.out‚29343.71044921875 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 391.67578125 disk.dm-1.out‚ 391.67578125) disk./.usagePercent‚17.42802855069752( disk.nvme0c0n1.out‚29343.71044921875" proc.memory.rssMB‚ 2772.3359375 network.recv‚ 2.513421e+06 disk.nvme0n1.in‚4.671875# disk.dm-0.out‚29343.71044921875 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1598.69140625 network.sent‚ 2.0562022e+07 disk.rbd1.in‚0V¾J¢Þ!‚(:’ †ÚäÔæì ~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷h~g`ß!‚(:Õ †ÚäÔÒžž& disk.nvme0c0n1.out‚29388.740234375 disk.sda2.out‚0$ disk.nvme0n1.out‚29388.740234375 disk.nvme0c0n1.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1598.69140625% disk./.usageGB‚259.69604110717773 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375! disk.dm-0.out‚29388.740234375 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.920995712280273 disk.sda2.in‚0 network.sent‚ 2.0670584e+07* disk./.usagePercent‚17.428419218194772 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2080.in‚0 network.recv‚ 2.520101e+06 disk.nvme0n1.in‚4.671875# memory_percent‚90.2423620223999 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875 disk.rbd1.out‚ 3.6796875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0" proc.memory.rssMB‚ 2772.3359375 cpu‚25.355977107291487 disk.drbd1716.in‚0 disk.dm-1.out‚392.5 disk.sda3.out‚392.5 disk.drbd2080.out‚0 disk.sda.out‚392.5fAåh2à!- _wandbruntime‚6943  _runtime‚6943(MhÉá!à _step‚709 ' lr-AdamW/pg1‚3.8311721727353784e-05 ' lr-AdamW/pg2‚3.8311721727353784e-05 trainer/global_step‚8649 % _timestamp‚1.7884234439637465e+09 _runtime‚6943.701308036•†'Îâ!È _step‚709 ' lr-AdamW/pg1‚3.8311721727353784e-05 ' lr-AdamW/pg2‚3.8311721727353784e-05 trainer/global_step‚8649 % _timestamp‚1.7884234439637465e+09 _runtime‚6943.701308036Å6|"2ã!- _wandbruntime‚6943  _runtime‚6943'$)ç'ä!áO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115014850 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609951268352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550002734580224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7607414737920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126011442 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41487684 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455189 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564262 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089644544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚497725693793130 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884234439644353e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557610149318144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126011444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550002710640128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156500992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435151 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503712741221888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830866432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6943.702231409 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7224733126506 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792rCjkg å!æO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚510964830866432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚503712741221888 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115014850 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557609951268352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚550002734580224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7607414737920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126011442 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126011442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41487684 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163455189 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7225387564262 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7252089644544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚497725693793130 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚490500960666624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884234439644353e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557610149318144 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163455189 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126011444 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚550002710640128 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚510964156500992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163435151 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚510964156500992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚503712741221888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7251433282048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289466633 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚510964830866432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73527166 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6943.702231409 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚503712723218944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7224733126506 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557672652575232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7608107841024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚711 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884234446709125e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115027078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7608267257344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126025458 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499129 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252252709888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41491746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115027519 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 6944.40923222 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503769076257792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561470Û ù õ&è!ïM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7252252709888 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289479063 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚550064385317888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚490555864797184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884234446709125e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115027078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚503769094260736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7608267257344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126025458 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126025460 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289499129 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163453605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7252252709888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚550064361377792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163473669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41491746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115027519 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚441 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 6944.40923222 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚503769076257792 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7225549561470 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289479063 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 163453605 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7252909085696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163473669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73535332 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7252909085696 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557672652575232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7608107841024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8649 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289499129 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚511022003346432 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚711 Fp’î!‚(:‡ ¤ÚäÔ”¢þ~ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.598' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7196 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„>bvï!‚(:ë ¤ÚäÔü¨ disk.drbd2265.in‚0" disk.dm-0.out‚29512.8662109375 disk.dm-1.out‚ 393.69921875) disk./.usagePercent‚17.42857563879819% disk./.usageGB‚259.69837188720703 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 cpu‚25.354698215934935 disk.rbd0.out‚0 network.recv‚ 2.530448e+06 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 network.sent‚ 2.0785454e+07 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.671875% disk.nvme0n1.out‚29512.8662109375 disk.nvme0n1.in‚4.671875' disk.nvme0c0n1.out‚29512.8662109375 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875" proc.memory.rssMB‚ 2772.3359375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375* proc.memory.percent‚16.920995712280273 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1598.65234375 disk.sda.out‚ 393.69921875 disk.sda2.out‚0 disk.sda3.out‚ 393.69921875" memory_percent‚90.242600440979 proc.cpu.threads‚17Ó2@ ð!‚(:… ³ÚäԮ͂~ gpu.‰d•p0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.54' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nž0rñ!‚(:ç ³ÚäÔêü€ disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.out‚0# memory_percent‚90.2425765991211* proc.memory.percent‚16.920995712280273 network.sent‚ 2.0790761e+07% disk./.usageGB‚259.72138595581055 disk.drbd2080.out‚0" disk.dm-0.out‚29552.9345703125 disk.dm-1.in‚ 126.77734375 disk.sda3.out‚ 394.06640625 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 394.06640625 proc.cpu.threads‚17' disk.nvme0c0n1.out‚29552.9345703125 network.recv‚ 2.534e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚29552.9345703125 disk.rbd0.in‚0 disk.nvme0n1.in‚4.671875 disk.nvme0c0n1.in‚4.671875 disk.sda.out‚ 394.06640625 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.6796875 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2772.3359375 cpu‚25.354012194062662* disk./.usagePercent‚17.430120132252263& proc.memory.availableMB‚ 1598.65625åŠ2ò!- _wandbruntime‚6981  _runtime‚6981Ç'Çó!Á trainer/global_step‚8699 % _timestamp‚1.7884234815224304e+09 _runtime‚6981.259999616 _step‚713 & lr-AdamW/pg1‚3.729628654823845e-05 & lr-AdamW/pg2‚3.729628654823845e-05Ø…tZÌô!Æ & lr-AdamW/pg1‚3.729628654823845e-05 & lr-AdamW/pg2‚3.729628654823845e-05 trainer/global_step‚8699 % _timestamp‚1.7884234815224304e+09 _runtime‚6981.259999616 _step‚713ÉæQ2õ!- _wandbruntime‚6981  _runtime‚6981%\Cç'ö!áO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚8699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560735634442752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚513823468614656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500511735660538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7650209841152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381241 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73935566 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164361203 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41713266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293075450368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41712843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560735457364480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 _step‚714 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884234815231252e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794249216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126712244 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553085247523328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7293075450368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚493246184233984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115648832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164381241 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872» ¸ì'÷!æO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115648407 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126712244 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚425 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884234815231252e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513822794249216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚513822794249216 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126712244 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚553085247523328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7293075450368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚423 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚493246184233984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 164361203 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚493246167194624 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115648832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 164381241 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7292419087872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 trainer/global_step‚8699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚560735634442752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚513823468614656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚500511735660538 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7650209841152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164381241 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7265568465914 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513823468614656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73935566 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164361203 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚553085271463424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73935564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126712242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506530375161344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41713266 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7293075450368 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41712843 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚560735457364480 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 153138176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚506530375161344 _step‚714 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797973247488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293238515712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41717748 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553146922201088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293238515712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41717317 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚8699 % _timestamp‚1.7884234822323563e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚715 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506586746203136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6981.970351156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115661482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7651234402816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115661049 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513879966715904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126726260 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126726258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506586746203136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894891520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚513880641094656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379657 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560798156603904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399721 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7267039338842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0.ÂMäú!ðM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126726260 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 12mÜ2"&6726258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚553146898260992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506586746203136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7293894891520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164379657 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚513880641094656 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚500567456226062 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513880641094656 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164399721 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164379657 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚560798156603904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506586728200192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291105915 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164399721 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚433 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7267039338842 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7293894891520 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291105915 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚560797973247488 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506586728200192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7293238515712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41717748 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚553146922201088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7293238515712 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41717317 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚500568127703386 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚8699 % _timestamp‚1.7884234822323563e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚715 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚493301088364544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506586746203136 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73943732 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291125981 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚6981.970351156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115661482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73943734 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7651234402816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291125981 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126726260 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115661049 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7266384900878 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513879966715904ËÒZ¤2û!- _wandbruntime‚6981  _runtime‚6981Z”,t]ü!× 2loss/SRqqt2/assignment_loss‚1.7609274131911141 2loss/SRqqt3/assignment_loss‚1.5289071503010663 1loss/SRqqt2/detection_loss‚2.2603873917034694 1loss/SRqqt3/detection_loss‚1.6730484669858758 .loss/FBt3/detection_loss‚3.396449327468872 /loss/FBt4/detection_loss‚0.5218500502717992 .loss/FRt3/detection_loss‚2.646397315329168 /loss/FRt4/detection_loss‚1.5145466613272827 epoch‚7 trainer/global_step‚8699 1loss/FBt3/assignment_loss‚0.24383241993685564 1loss/SRqqt4/detection_loss‚0.9421989004780997 % _timestamp‚1.7884234822342722e+09 _runtime‚6981.971706824 0loss/FRt3/assignment_loss‚2.5116913071492823 %loss/total_loss‚41.70046948007013 0loss/FRt4/assignment_loss‚1.1382973743420022 2loss/SRqqt4/assignment_loss‚0.8795723322007472 0loss/SRqqt1/detection_loss‚1.904131916436282 _step‚716 /loss/FRt1/assignment_loss‚3.265236243313434 0loss/FRt2/assignment_loss‚2.8865726305090864 /loss/FBt1/detection_loss‚2.4029362112283708 /loss/FBt2/detection_loss‚2.5594963908195494 .loss/FRt1/detection_loss‚2.334279471168331 2loss/SRqqt1/assignment_loss‚1.5316001490097155 1loss/FBt1/assignment_loss‚0.17521488883678557 1loss/FBt2/assignment_loss‚0.35567037634318693 1loss/FBt4/assignment_loss‚0.12326848491702411 .loss/FRt2/detection_loss‚3.143956606802733ñÙàbý!Ü 1loss/FBt3/assignment_loss‚0.24383241993685564 1loss/SRqqt4/detection_loss‚0.9421989004780997 % _timestamp‚1.7884234822342722e+09 _runtime‚6981.971706824 0loss/FRt3/assignment_loss‚2.5116913071492823 %loss/total_loss‚41.70046948007013 0loss/FRt4/assignment_loss‚1.1382973743420022 2loss/SRqqt4/assignment_loss‚0.8795723322007472 0loss/SRqqt1/detection_loss‚1.904131916436282 _step‚716 /loss/FRt1/assignment_loss‚3.265236243313434 0loss/FRt2/assignment_loss‚2.8865726305090864 /loss/FBt1/detection_loss‚2.4029362112283708 /loss/FBt2/detection_loss‚2.5594963908195494 .loss/FRt1/detection_loss‚2.334279471168331 2loss/SRqqt1/assignment_loss‚1.5316001490097155 1loss/FBt1/assignment_loss‚0.17521488883678557 1loss/FBt2/assignment_loss‚0.35567037634318693 1loss/FBt4/assignment_loss‚0.12326848491702411 .loss/FRt2/detection_loss‚3.143956606802733 2loss/SRqqt2/assignment_loss‚1.7609274131911141 2loss/SRqqt3/assignment_loss‚1.5289071503010663 1loss/SRqqt2/detection_loss‚2.2603873917034694 1loss/SRqqt3/detection_loss‚1.6730484669858758 .loss/FBt3/detection_loss‚3.396449327468872 /loss/FBt4/detection_loss‚0.5218500502717992 .loss/FRt3/detection_loss‚2.646397315329168 /loss/FRt4/detection_loss‚1.5145466613272827 epoch‚7 trainer/global_step‚8699Ì$áN ’þ!‚(:‡ ÂÚäÔ‚Ò½~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·mò¿xÿ!‚(:í ÂÚäԿļ$ disk./.usageGB‚259.6857604980469 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4.671875 disk.sda1.in‚0) proc.memory.availableMB‚ 1124.09765625" proc.memory.rssMB‚ 2772.3359375 cpu‚25.353358699835553 disk.drbd1716.in‚0% disk.nvme0n1.out‚29605.4345703125' disk.nvme0c0n1.out‚29605.4345703125 disk.sda3.out‚ 394.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0$ memory_percent‚93.13905239105225* proc.memory.percent‚16.920995712280273 network.sent‚ 2.0796847e+07 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 394.77734375 network.recv‚ 2.538403e+06 disk.nvme0n1.in‚4.671875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375" disk.dm-0.out‚29605.4345703125 disk.drbd2266.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚17.427729277889995 disk.sda2.out‚0 disk.rbd1.out‚ 3.6796875 disk.nvme0c0n1.in‚4.671875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 394.77734375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 proc.cpu.threads‚17Zª i€"‚(:’ ÑÚäÔ¬»ž~ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.255' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.050999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lê)l"‚(:á ÑÚäÔ¤þ“ disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0* proc.memory.percent‚16.920995712280273 proc.cpu.threads‚17 disk.rbd1.out‚ 3.6796875 network.sent‚ 2.0905318e+07$ disk./.usageGB‚259.7303047180176 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 395.1796875 disk.dm-0.in‚4.671875 disk.drbd1716.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚17.43071867786732 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0$ memory_percent‚93.13902854919434 disk.rbd0.in‚0% disk.nvme0c0n1.out‚29665.69140625 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.dm-0.out‚29665.69140625 disk.sda.out‚ 395.1796875" proc.memory.rssMB‚ 2772.3359375 disk.nvme0n1.in‚4.671875# disk.nvme0n1.out‚29665.69140625 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 395.1796875( proc.memory.availableMB‚ 1124.1015625 network.recv‚ 2.54423e+06 disk.nvme0c0n1.in‚4.671875 cpu‚25.352678333430124‡,Ù2‚"- _wandbruntime‚7018  _runtime‚7018<ÌÝóǃ"Á _step‚717 & lr-AdamW/pg1‚3.629065589691725e-05 & lr-AdamW/pg2‚3.629065589691725e-05 trainer/global_step‚8749 % _timestamp‚1.7884235183904898e+09 _runtime‚7018.1280099170»{Ì„"Æ trainer/global_step‚8749 % _timestamp‚1.7884235183904898e+09 _runtime‚7018.128009917 _step‚717 & lr-AdamW/pg1‚3.629065589691725e-05 & lr-AdamW/pg2‚3.629065589691725e-05Í|íÃ2…"- _wandbruntime‚7018  _runtime‚70184;?¾æ'†"àO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚516682106362880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚509348027103744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7334061256192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503297777527946 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431997440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41939301 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165307293 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165287255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚509348027103744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292700297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516682106362880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚495991373722624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127413044 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165287255 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333404893696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788423518391207e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556167808346624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333404893696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7693242440192 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚516681431997440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165287255 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚509348045106688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333404893696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 $ _timestamp‚1.788423518391207e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚556167808346624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333404893696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚509348045106688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7693242440192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7334061256192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503297777527946 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚516681431997440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127413042 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41939301 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165307293 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165287255 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚509348027103744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292700297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚516682106362880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443Îþ3½2ˆ"- _wandbruntime‚7018  _runtime‚7018 §Oñ&‰"ëM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7694270720000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116295909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503354169570794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7307220240286 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41944208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884235190789456e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚509404380142592 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127427058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563923729804288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556229459084288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41943777 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732767 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516738604464128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880697344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307874678250 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334880697344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563923552739328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732767 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116296342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¡w…ö&Š"ðM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503353498093470 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165325773 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 % _timestamp‚1.7884235190789456e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚509404380142592 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127427058 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚563923729804288 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚556229459084288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165305709 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41943777 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292732767 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚719 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚516738604464128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚509404398145536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334880697344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307874678250 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127427060 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334880697344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74352132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563923552739328 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292732767 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚509404398145536 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116296342 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚431 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7694270720000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116295909 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74352134 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503354169570794 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7307220240286 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292752833 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚496046294892544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚496046277853184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41944208ÏŠnª÷2‹"- _wandbruntime‚7018  _runtime‚7018‡ãD¢[Œ"Õ .loss/FRt3/detection_loss‚2.137676081920074 /loss/FBt2/detection_loss‚2.7225080609321592 trainer/global_step‚8749 _step‚720 /loss/FRt3/assignment_loss‚2.876250171055228 1loss/FBt3/assignment_loss‚0.19115508114919066 /loss/FRt2/detection_loss‚2.7616584519545238 %loss/total_loss‚43.06346350792152 2loss/SRqqt1/assignment_loss‚1.2485849536024034 2loss/SRqqt2/assignment_loss‚2.1938223442242992 2loss/SRqqt4/assignment_loss‚1.1081639258579041 .loss/FRt1/detection_loss‚2.591673536880596 0loss/FRt2/assignment_loss‚3.0703664422035217 1loss/SRqqt3/detection_loss‚1.6552847978328504 epoch‚7 _runtime‚7018.818780656 0loss/FBt2/assignment_loss‚0.1350593661190942 /loss/FRt4/detection_loss‚1.6104999115147318 1loss/SRqqt4/detection_loss‚0.8706519303448272 .loss/FBt3/detection_loss‚4.288969039916992 /loss/FBt4/detection_loss‚0.5241505211399106 2loss/SRqqt3/assignment_loss‚1.0328942912240182 1loss/FBt1/assignment_loss‚0.42574174888432026 .loss/FBt1/detection_loss‚2.169054025953466 1loss/SRqqt1/detection_loss‚2.2483060965314507 1loss/SRqqt2/detection_loss‚2.1947688056576635 % _timestamp‚1.7884235190813203e+09 /loss/FRt1/assignment_loss‚3.569342268479837 /loss/FRt4/assignment_loss‚1.288523965825637 1loss/FBt4/assignment_loss‚0.14835768871682087€ ¢Z`"Ú /loss/FRt3/assignment_loss‚2.876250171055228 1loss/FBt3/assignment_loss‚0.19115508114919066 /loss/FRt2/detection_loss‚2.7616584519545238 %loss/total_loss‚43.06346350792152 2loss/SRqqt1/assignment_loss‚1.2485849536024034 2loss/SRqqt2/assignment_loss‚2.1938223442242992 2loss/SRqqt4/assignment_loss‚1.1081639258579041 .loss/FRt1/detection_loss‚2.591673536880596 0loss/FRt2/assignment_loss‚3.0703664422035217 1loss/SRqqt3/detection_loss‚1.6552847978328504 epoch‚7 _runtime‚7018.818780656 0loss/FBt2/assignment_loss‚0.1350593661190942 /loss/FRt4/detection_loss‚1.6104999115147318 1loss/SRqqt4/detection_loss‚0.8706519303448272 .loss/FBt3/detection_loss‚4.288969039916992 /loss/FBt4/detection_loss‚0.5241505211399106 2loss/SRqqt3/assignment_loss‚1.0328942912240182 1loss/FBt1/assignment_loss‚0.42574174888432026 .loss/FBt1/detection_loss‚2.169054025953466 1loss/SRqqt1/detection_loss‚2.2483060965314507 1loss/SRqqt2/detection_loss‚2.1947688056576635 % _timestamp‚1.7884235190813203e+09 /loss/FRt1/assignment_loss‚3.569342268479837 /loss/FRt4/assignment_loss‚1.288523965825637 1loss/FBt4/assignment_loss‚0.14835768871682087 .loss/FRt3/detection_loss‚2.137676081920074 /loss/FBt2/detection_loss‚2.7225080609321592 trainer/global_step‚8749 _step‚720ÐÑ W»‘Ž"‚(:† àÚäÔ÷º¥~ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.991 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l©]Úv"‚(:ë àÚäÔËÃÁ€ network.sent‚ 2.0911719e+07 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.920995712280273* disk./.usagePercent‚17.430765527246084 disk.drbd2080.out‚0 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚259.7310028076172 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚29708.81787109375( disk.nvme0c0n1.out‚29708.81787109375 disk.sda2.out‚0$ memory_percent‚93.14079284667969 network.recv‚ 2.549352e+06 disk.sda1.out‚0# disk.dm-0.out‚29708.81787109375 disk.drbd2266.in‚0 disk.sda.out‚ 395.84765625 disk.drbd2265.out‚0% proc.memory.availableMB‚ 1123.8125 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 395.84765625 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.dm-1.out‚ 395.84765625 disk.rbd1.out‚ 3.6796875" proc.memory.rssMB‚ 2772.3359375 cpu‚25.35207420006103 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875¯U %"‚(:’ ïÚäÔÊö} gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.938' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.187599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œRkw‘"‚(:ì ïÚäÔâÚþ disk.rbd1.in‚0 disk.sda.out‚ 396.19921875 disk.nvme0n1.in‚4.671875& disk.nvme0n1.out‚29767.58251953125( disk.nvme0c0n1.out‚29767.58251953125 disk.sda3.out‚ 396.19921875 disk.drbd2080.in‚0# disk.dm-0.out‚29767.58251953125 disk.rbd1.out‚ 3.6796875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0$ memory_percent‚93.13926696777344 proc.cpu.threads‚17 disk.dm-0.in‚4.671875 disk.drbd2266.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2772.3359375 network.recv‚ 2.554823e+06 disk.drbd2266.out‚0 disk.rbd0.out‚0* disk./.usagePercent‚17.431938041752904 disk.drbd1716.out‚0* proc.memory.percent‚16.920995712280273 network.sent‚ 2.1019776e+07% disk./.usageGB‚259.74847412109375 disk.dm-1.out‚ 396.19921875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0% proc.memory.availableMB‚ 1124.0625 cpu‚25.351401042782133¬tü’"‚(:… þÚäÔÓ†~ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ßVÎ{“"‚(:ð þÚäÔÉŸ’€$ memory_percent‚93.13781261444092* proc.memory.percent‚16.920995712280273 cpu‚25.350761030161586 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚29815.86767578125& disk.nvme0n1.out‚29815.86767578125 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 396.88671875 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1124.30078125 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0) disk./.usagePercent‚17.43268814382823# disk.dm-0.out‚29815.86767578125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.1025733e+07 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 396.88671875 disk.nvme0n1.in‚4.671875% disk./.usageGB‚259.75965118408203 disk.drbd2080.in‚0 disk.dm-1.out‚ 396.88671875 disk.sda1.in‚0" proc.memory.rssMB‚ 2772.3359375 network.recv‚ 2.559275e+06 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.drbd2265.out‚0½¥íj2”"- _wandbruntime‚7055  _runtime‚7055õ¹NÈ•" _runtime‚ 7055.24755283 _step‚721 ' lr-AdamW/pg1‚3.5295039345310674e-05 ' lr-AdamW/pg2‚3.5295039345310674e-05 trainer/global_step‚8799 % _timestamp‚1.7884235555100362e+09nØþÎÍ–"Ç _runtime‚ 7055.24755283 _step‚721 ' lr-AdamW/pg1‚3.5295039345310674e-05 ' lr-AdamW/pg2‚3.5295039345310674e-05 trainer/global_step‚8799 % _timestamp‚1.7884235555100362e+09ÑÉÒt 2—"- _wandbruntime‚7055  _runtime‚7055ø¨t˜"áO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047062016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519540744111104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7374390699520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519540069745664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42164609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512165679046144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166213307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42164180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 % _timestamp‚1.7884235555107098e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166233345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559250321289728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7736236914688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7347893582486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7347239144730 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7375047062016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566986733185536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128113842 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128113844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7736387955712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116916975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7375047062016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519540744111104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 151041024 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7374390699520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519540069745664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚512165679046144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519540069745664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42164609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294327149 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚512165679046144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166213307 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74752364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42164180 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚559250345229824 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498736580250624 % _timestamp‚1.7884235555107098e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166233345 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚559250321289728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚431 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74752366 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7736236914688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 174981120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7347893582486 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7347239144730 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7375047062016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚566986733185536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128113842 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128113844 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚506084490872470 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7736387955712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚512165697049088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166213307 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7374390699520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294347189 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚519540744111104 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116916975 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚429 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884235561987758e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7348055579694 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017658 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916591104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116929240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7737268050432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359619 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7375866503168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242212352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42168708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559311972027392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128127860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866503168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567049086952960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7375210127360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8799 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519597916591104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚506140211438202 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 % _timestamp‚1.7884235561987758e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498791484381184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7348055579694 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7348710017658 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519597916591104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294379685 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116929240 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74760532 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocaÁI„òted‚ 7737268050432 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294359619 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚512222032084992 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506139539960878 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294359619 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚512222050087936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7375866503168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519597242212352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166231761 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42168708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚723 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚559311972027392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128127860 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7375866503168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128127860 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166251825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294379685 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚567049086952960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚512222032084992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7375210127360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8799 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032867840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032867840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7779257092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501481803817984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42391586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128814644 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788423592083746e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167159397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570112300731904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167139359 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570112115265024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚508870532739878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117552352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42391153 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514983330988544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779418618880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¥½ûhä#©"åO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚522398707493888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128814644 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295954001 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167139359 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514983348991488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚522399381859328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7415376505344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚522398707493888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514983330988544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75160764 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7779418618880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚433 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75160766 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128814642 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7416032867840 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514983330988544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7416032867840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7779257092096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚501481803817984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128814642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚522399381859328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42391586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562332858172928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128814644 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788423592083746e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167159397 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚570112300731904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 295974041 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7388728921894 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167159397 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514983348991488 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167139359 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚570112115265024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requestžàâed_bytes.all.allocated‚508870532739878 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117552352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42391153 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7780282127872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117564595 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501536690909184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117565030 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884235927674384e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42395663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879960576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42396096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515039684027392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780122711552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195933184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570174814978560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚8849 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0¤sö&¬"ðM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚515039702030336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚515039684027392 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚508925581828286 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7388890919102 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128828660 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7780122711552 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75168932 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7416195933184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 296006537 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚522456554339328 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚570174814978560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚8849 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7416852308992 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7416195933184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7389545357066 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295986471 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128828660 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7780282127872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296006537 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117564595 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚501536690909184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117565030 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562394532850688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884235927674384e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128828658 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42395663 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128828658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562394508910592 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚522455879960576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75168934 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42396096ׇô}Ò2­"- _wandbruntime‚7092  _runtime‚7092ÅòÄhY®"Ó epoch‚7 _step‚728 0loss/FRt3/assignment_loss‚2.7638317969712345 /loss/FBt4/detection_loss‚0.5238512732699369 0loss/FRt4/assignment_loss‚0.9712514690324372 -loss/FRt2/detection_loss‚3.03971090581682 1loss/SRqqt4/detection_loss‚0.9595287960309249 _runtime‚7092.506480322 %loss/total_loss‚42.41976931921005 /loss/FRt1/assignment_loss‚3.256695198769472 2loss/SRqqt2/assignment_loss‚1.5849523428947694 /loss/FRt1/detection_loss‚2.1743843926458943 0loss/SRqqt2/detection_loss‚2.581068063935926 /loss/FBt3/detection_loss‚2.9178878366947174 % _timestamp‚1.7884235927690606e+09 2loss/SRqqt1/assignment_loss‚1.2955916825855194 1loss/SRqqt4/assignment_loss‚1.062575994202724 0loss/FBt1/assignment_loss‚0.0518319197422395 /loss/FBt2/assignment_loss‚1.046756535768509 trainer/global_step‚8849 2loss/FBt3/assignment_loss‚0.029722570364053052 .loss/FBt2/detection_loss‚4.185787200927734 2loss/SRqqt3/assignment_loss‚1.3007651321847546 1loss/FBt4/assignment_loss‚0.09251287062584411 /loss/FRt3/detection_loss‚2.4836766953495415 /loss/FRt4/detection_loss‚1.5874208980696458 .loss/FBt1/detection_loss‚1.729830327359113 0loss/FRt2/assignment_loss‚3.2488947091279208 1loss/SRqqt1/detection_loss‚1.7256253316651944 0loss/SRqqt3/detection_loss‚1.805615375175121ExÎÛ^¯"Ø 0loss/FRt2/assignment_loss‚3.2488947091279208 1loss/SRqqt1/detection_loss‚1.7256253316651944 0loss/SRqqt3/detection_loss‚1.805615375175121 epoch‚7 _step‚728 0loss/FRt3/assignment_loss‚2.7638317969712345 /loss/FBt4/detection_loss‚0.5238512732699369 0loss/FRt4/assignment_loss‚0.9712514690324372 -loss/FRt2/detection_loss‚3.03971090581682 1loss/SRqqt4/detection_loss‚0.9595287960309249 _runtime‚7092.506480322 %loss/total_loss‚42.41976931921005 /loss/FRt1/assignment_loss‚3.256695198769472 2loss/SRqqt2/assignment_loss‚1.5849523428947694 /loss/FRt1/detection_loss‚2.1743843926458943 0loss/SRqqt2/detection_loss‚2.581068063935926 /loss/FBt3/detection_loss‚2.9178878366947174 % _timestamp‚1.7884235927690606e+09 2loss/SRqqt1/assignment_loss‚1.2955916825855194 1loss/SRqqt4/assignment_loss‚1.062575994202724 0loss/FBt1/assignment_loss‚0.0518319197422395 /loss/FBt2/assignment_loss‚1.046756535768509 trainer/global_step‚8849 2loss/FBt3/assignment_loss‚0.029722570364053052 .loss/FBt2/detection_loss‚4.185787200927734 2loss/SRqqt3/assignment_loss‚1.3007651321847546 1loss/FBt4/assignment_loss‚0.09251287062584411 /loss/FRt3/detection_loss‚2.4836766953495415 /loss/FRt4/detection_loss‚1.5874208980696458 .loss/FBt1/detection_loss‚1.729830327359113ØæÇç\’°"‚(:‡ «ÛäÔç÷—~ gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ³Zˆx±"‚(:í «ÛäÔ¬›Š disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 cpu‚25.348853573350734 disk.rbd0.out‚0 disk.dm-1.out‚ 398.41015625 disk.drbd2080.in‚0# disk.dm-0.out‚29991.89794921875 disk.rbd1.out‚ 3.6796875 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.671875 disk.sda3.out‚ 398.41015625 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚29991.89794921875 disk.rbd0.in‚0$ memory_percent‚93.13983917236328& proc.memory.availableMB‚ 1123.96875" proc.memory.rssMB‚ 2772.3359375 disk.sda.out‚ 398.41015625 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17 disk.dm-0.in‚4.671875 disk.nvme0c0n1.in‚4.671875* proc.memory.percent‚16.920995712280273 disk.drbd2266.out‚0 network.sent‚ 2.1145832e+07* disk./.usagePercent‚17.433719854191214 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚29991.89794921875 network.recv‚ 2.573056e+06$ disk./.usageGB‚259.7750244140625ý®Àj’²"‚(:‡ ºÛäÔ¾Äð~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚180.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­TÏæu³"‚(:ê ºÛäÔÒ²µ disk.nvme0c0n1.in‚4.671875( disk.nvme0c0n1.out‚30045.35107421875 disk.drbd2080.in‚0 disk.dm-0.in‚4.671875 disk.sda2.in‚0" proc.memory.rssMB‚ 2772.3359375* proc.memory.percent‚16.920995712280273 cpu‚25.348222702408098% disk./.usageGB‚259.80209732055664 disk.drbd1716.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚30045.35107421875 proc.cpu.threads‚17 network.recv‚ 2.579105e+06) disk./.usagePercent‚17.43553673966172 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 399.078125 disk.sda.out‚ 399.078125 disk.sda1.out‚0 disk.sda2.out‚0 network.sent‚ 2.1254358e+07 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.6796875$ memory_percent‚93.14286708831787 disk.dm-1.out‚ 399.078125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0) proc.memory.availableMB‚ 1123.47265625 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2265.in‚0# disk.dm-0.out‚30045.35107421875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.671875âœÛ:´"‚(:’ ÉÛäÔ¹äÇ~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚142.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.546599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›³("lµ"‚(:á ÉÛäÔÔÙÞ€ disk.drbd1716.in‚0 disk.sda.out‚399.625$ memory_percent‚93.13986301422119 disk.drbd2265.in‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1123.96484375 disk.nvme0n1.in‚4.671875( disk.nvme0c0n1.out‚30093.88037109375 disk.sda3.out‚399.625 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚399.625 disk.rbd1.out‚ 3.6796875& disk.nvme0n1.out‚30093.88037109375 disk.rbd0.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚30093.88037109375 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda3.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375) disk./.usagePercent‚17.43756969549112 disk.drbd2080.out‚0 disk.dm-0.in‚4.671875 disk.sda1.in‚0" proc.memory.rssMB‚ 2772.3359375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 network.recv‚ 2.582273e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.920995712280273 cpu‚25.347602217229507 network.sent‚ 2.1259436e+07% disk./.usageGB‚259.83238983154297'Ž½2¶"- _wandbruntime‚7128  _runtime‚7128 ðGÈ·" _step‚729 ' lr-AdamW/pg1‚3.3334676351066844e-05 ' lr-AdamW/pg2‚3.3334676351066844e-05 trainer/global_step‚8899 $ _timestamp‚1.788423628775705e+09 _runtime‚7128.513218191Å"*ñ͸"Ç _runtime‚7128.513218191 _step‚729 ' lr-AdamW/pg1‚3.3334676351066844e-05 ' lr-AdamW/pg2‚3.3334676351066844e-05 trainer/global_step‚8899 $ _timestamp‚1.788423628775705e+09Ùü‹2¹"- _wandbruntime‚7128  _runtime‚7128Ã}“Qfº"áO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573237814973952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823546 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129515444 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 % _timestamp‚1.7884236287763758e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚7128.514115614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚730 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019607552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118187936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565415395056128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/alloc2—ñation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7822395977728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7456362311168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362311168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75569166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42619229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚8899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345242112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118188395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573237619021312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7822223965184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚517801000933888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42618772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944¥Œì'»"æO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525258019607552 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525257345242112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7457018673664 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚573237814973952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7428909823546 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129515444 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 % _timestamp‚1.7884236287763758e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7457018673664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚7128.514115614 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚504226993306624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚730 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129515442 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚511656574607286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525258019607552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118187936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75569164 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚565415395056128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚504227010345984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚511655903130170 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297600893 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297580853 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7822395977728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7456362311168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7456362311168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75569166 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42619229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 trainer/global_step‚8899 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525257345242112 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118188395 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚517801000933888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚573237619021312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168085449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7822223965184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚517800982930944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚517801000933888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42618772 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7429564261302 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129515442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚459 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168085449 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚517800982930944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75577332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42623321 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129529460 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192087552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884236294636624e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚731 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168103929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118200653 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118201108 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103929 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792Ñ&AM¾"ðM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚565477069733888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75577332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42623321 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚565477045793792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚8899 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_poolý‚Oú©.allocated‚ 129529460 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚504281914476544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚525315192087552 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168083865 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚511712295173018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884236294636624e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _step‚731 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168103929 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚517857353972736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118200653 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚511711623695694 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118201108 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297633389 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129529460 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168103929 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚517857335969792 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297613323 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75577334 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42623774 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30ÛÄnQ2¿"- _wandbruntime‚7129  _runtime‚7129çíø^À"Ø 2loss/SRqqt2/assignment_loss‚1.7883674541586323 2loss/SRqqt3/assignment_loss‚1.3299558807026457 1loss/SRqqt3/detection_loss‚1.9485753667023447 &loss/total_loss‚41.687462596802966 0loss/FRt4/assignment_loss‚0.9624150884151459 /loss/FRt4/detection_loss‚1.7396176604926585 /loss/FBt4/detection_loss‚0.6085776756425683 trainer/global_step‚8899 _step‚732 1loss/SRqqt1/assignment_loss‚1.473604382680995 /loss/FRt3/detection_loss‚2.4329119266533272 0loss/FRt3/assignment_loss‚2.9139544847534924 1loss/SRqqt4/assignment_loss‚0.827795822222374 1loss/FBt4/assignment_loss‚0.12944409959088254 1loss/SRqqt1/detection_loss‚2.0284310153552463 0loss/SRqqt2/detection_loss‚2.238232076952332 % _timestamp‚1.7884236294652884e+09 1loss/FBt2/assignment_loss‚0.25320320920679606 1loss/FBt3/assignment_loss‚0.21540757751790807 .loss/FBt3/detection_loss‚2.537155909197671 _runtime‚7129.202707665 /loss/FRt2/assignment_loss‚3.323635810095331 /loss/FRt1/detection_loss‚2.3508956618607044 /loss/FRt2/detection_loss‚2.5744385524936346 /loss/FBt1/detection_loss‚3.1370176672935486 /loss/FBt2/detection_loss‚2.8477982878684998 epoch‚7 0loss/FRt1/assignment_loss‚2.4637115386625132 0loss/FBt1/assignment_loss‚0.5909575742506215 1loss/SRqqt4/detection_loss‚0.9713578740330903¡ ;,cÁ"Ý 0loss/FRt1/assignment_loss‚2.4637115386625132 0loss/FBt1/assignment_loss‚0.5909575742506215 1loss/SRqqt4/detection_loss‚0.9713578740330903 2loss/SRqqt2/assignment_loss‚1.7883674541586323 2loss/SRqqt3/assignment_loss‚1.3299558807026457 1loss/SRqqt3/detection_loss‚1.9485753667023447 &loss/total_loss‚41.687462596802966 0loss/FRt4/assignment_loss‚0.9624150884151459 /loss/FRt4/detection_loss‚1.7396176604926585 /loss/FBt4/detection_loss‚0.6085776756425683 trainer/global_step‚8899 _step‚732 1loss/SRqqt1/assignment_loss‚1.473604382680995 /loss/FRt3/detection_loss‚2.4329119266533272 0loss/FRt3/assignment_loss‚2.9139544847534924 1loss/SRqqt4/assignment_loss‚0.827795822222374 1loss/FBt4/assignment_loss‚0.12944409959088254 1loss/SRqqt1/detection_loss‚2.0284310153552463 0loss/SRqqt2/detection_loss‚2.238232076952332 % _timestamp‚1.7884236294652884e+09 1loss/FBt2/assignment_loss‚0.25320320920679606 1loss/FBt3/assignment_loss‚0.21540757751790807 .loss/FBt3/detection_loss‚2.537155909197671 _runtime‚7129.202707665 /loss/FRt2/assignment_loss‚3.323635810095331 /loss/FRt1/detection_loss‚2.3508956618607044 /loss/FRt2/detection_loss‚2.5744385524936346 /loss/FBt1/detection_loss‚3.1370176672935486 /loss/FBt2/detection_loss‚2.8477982878684998 epoch‚7Ü+¡Ž¾’Â"‚(:‡ ØÛäÔæ¡¿~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z/æ¯sÃ"‚(:è ØÛäÔ¡µÑ€) disk./.usagePercent‚17.43956681026573 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.671875# disk.dm-0.out‚30143.81005859375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚30143.81005859375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 400.2421875 network.recv‚ 2.586644e+06 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375& proc.memory.availableMB‚ 1123.90625 cpu‚25.34694518006466 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2772.3359375* proc.memory.percent‚16.920995712280273 network.sent‚ 2.1265563e+07& disk.nvme0n1.out‚30143.81005859375 disk.rbd0.in‚0 disk.dm-0.in‚4.671875 disk.nvme0n1.in‚4.671875 disk.sda3.out‚ 400.2421875 disk.drbd2266.out‚0 disk.rbd1.out‚ 3.6796875 disk.sda.out‚ 400.2421875$ memory_percent‚93.14022064208984 proc.cpu.threads‚17$ disk./.usageGB‚259.8621482849121 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda3.in‚ 126.77734375* “’Ä"‚(:‡ çÛäÔêø ~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚130.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ÀÎ {Å"‚(:ð çÛäԶ媀 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1123.90234375 proc.cpu.threads‚17 network.sent‚ 2.1373801e+07 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.671875( disk.nvme0c0n1.out‚30208.14697265625 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0 cpu‚25.346327146271165% disk./.usageGB‚259.86285400390625 disk.drbd2080.out‚0# disk.dm-0.out‚30208.14697265625 disk.rbd1.out‚ 3.6796875 disk.sda1.in‚0 disk.dm-0.in‚4.671875 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0* disk./.usagePercent‚17.439614171659564 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 400.64453125 disk.drbd2266.out‚0 disk.sda.out‚ 400.64453125& disk.nvme0n1.out‚30208.14697265625$ memory_percent‚93.14024448394775 network.recv‚ 2.59258e+06 disk.sda3.out‚ 400.64453125" proc.memory.rssMB‚ 2772.3359375* proc.memory.percent‚16.920995712280273 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0G…&~2Æ"- _wandbruntime‚7165  _runtime‚7165í˜ÅÇ"¿ % lr-AdamW/pg1‚3.23703384451393e-05 % lr-AdamW/pg2‚3.23703384451393e-05 trainer/global_step‚8949 % _timestamp‚1.7884236655859556e+09 _runtime‚7165.323505514 _step‚733%¡êCÊÈ"Ä _runtime‚7165.323505514 _step‚733 % lr-AdamW/pg1‚3.23703384451393e-05 % lr-AdamW/pg2‚3.23703384451393e-05 trainer/global_step‚8949 % _timestamp‚1.7884236655859556e+09Ý¢)«Å2É"- _wandbruntime‚7165  _runtime‚7165wÛ Ïç'Ê"áO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 337538560 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚568497931939328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚514442616474694 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130216244 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884236655866542e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42846500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚528115982990336 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118824066 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75977564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576363218258944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8949 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42846049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118823613 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576363032792064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7165.324392037 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991463 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 _step‚734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162954 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130216244 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚528116657355776 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7498004479488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚520618634873344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884236655866542e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42846500 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚528115982990336 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118824066 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75977564 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚576363218258944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚528115982990336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506972199834624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚8949 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7497348116992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42846049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130216242 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118823613 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75977566 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 168991463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚576363032792064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚520618652876288 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚7165.324392037 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168991463 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506972216873984 _step‚734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7469745162954 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ÞN¬ ,2Ì"- _wandbruntime‚7166  _runtime‚7166\NVxêÍ"ëM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118836758 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520674987912192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240175 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚7166.005995257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576425533036544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>Deviªqà ceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7498167544832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169009917 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42851024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118836301 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚528173829835776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 _step‚735 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884236662674756e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75985734 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42850569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507027103965184Iþ—ö&Î"ðM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169009917 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚528173829835776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7470561597918 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚514497665563102 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507027121004544 _step‚735 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884236662674756e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130230258 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169029981 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚528173829835776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7498823920640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471216035882 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169029981 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75985734 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42850569 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507027103965184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118836758 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚520674987912192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299240175 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚7166.005995257 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚520675005915136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚576425533036544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75985732 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299240175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514498337040426 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299260241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚520675005915136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7498167544832 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169009917 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42851024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚520674987912192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118836301ß:Û9f2Ï"- _wandbruntime‚7166  _runtime‚7166âgZ¤XÐ"Ò 1loss/SRqqt1/assignment_loss‚1.553346141999544 0loss/SRqqt2/detection_loss‚2.419894576072693 0loss/SRqqt4/detection_loss‚0.950381150158743 epoch‚7 trainer/global_step‚8949 _step‚736 2loss/SRqqt3/assignment_loss‚1.1603618473010628 1loss/FBt1/assignment_loss‚0.47633336830352035 0loss/FBt4/assignment_loss‚0.1320966778336415 /loss/FRt3/detection_loss‚1.9362490939891945 1loss/SRqqt3/detection_loss‚1.6151007217796225 2loss/SRqqt4/assignment_loss‚1.1613234313942182 1loss/FBt3/assignment_loss‚0.01161560098019739 0loss/SRqqt1/detection_loss‚2.203278793498527 _runtime‚ 7166.00716262 /loss/FRt1/detection_loss‚2.2884877066720617 /loss/FRt2/detection_loss‚3.1960918081217797 /loss/FRt4/detection_loss‚1.7597314507207449 % _timestamp‚1.7884236662694232e+09 0loss/FRt1/assignment_loss‚3.2041389508680864 0loss/SRqqt2/assignment_loss‚1.97576538225015 1loss/FBt2/assignment_loss‚0.18014804332051354 0loss/FRt4/assignment_loss‚1.0552116235097249 /loss/FBt4/detection_loss‚0.6021609179733787 %loss/total_loss‚43.03603355503353 0loss/FRt2/assignment_loss‚3.3207321906911917 0loss/FRt3/assignment_loss‚2.6686644645060524 .loss/FBt1/detection_loss‚2.918622170175825 -loss/FBt2/detection_loss‚2.67105313539505 /loss/FBt3/detection_loss‚3.5752443075180054xg¿F]Ñ"× /loss/FRt1/detection_loss‚2.2884877066720617 /loss/FRt2/detection_loss‚3.1960918081217797 /loss/FRt4/detection_loss‚1.7597314507207449 % _timestamp‚1.7884236662694232e+09 0loss/FRt1/assignment_loss‚3.2041389508680864 0loss/SRqqt2/assignment_loss‚1.97576538225015 1loss/FBt2/assignment_loss‚0.18014804332051354 0loss/FRt4/assignment_loss‚1.0552116235097249 /loss/FBt4/detection_loss‚0.6021609179733787 %loss/total_loss‚43.03603355503353 0loss/FRt2/assignment_loss‚3.3207321906911917 0loss/FRt3/assignment_loss‚2.6686644645060524 .loss/FBt1/detection_loss‚2.918622170175825 -loss/FBt2/detection_loss‚2.67105313539505 /loss/FBt3/detection_loss‚3.5752443075180054 1loss/SRqqt1/assignment_loss‚1.553346141999544 0loss/SRqqt2/detection_loss‚2.419894576072693 0loss/SRqqt4/detection_loss‚0.950381150158743 epoch‚7 trainer/global_step‚8949 _step‚736 2loss/SRqqt3/assignment_loss‚1.1603618473010628 1loss/FBt1/assignment_loss‚0.47633336830352035 0loss/FBt4/assignment_loss‚0.1320966778336415 /loss/FRt3/detection_loss‚1.9362490939891945 1loss/SRqqt3/detection_loss‚1.6151007217796225 2loss/SRqqt4/assignment_loss‚1.1613234313942182 1loss/FBt3/assignment_loss‚0.01161560098019739 0loss/SRqqt1/detection_loss‚2.203278793498527 _runtime‚ 7166.00716262à0 Œ•Ò"‚(:… öÛäÔøê¶~ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰0 zÓ"‚(:ï öÛäÔ˜¢ß€ disk.dm-0.in‚4.671875 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 401.33984375& disk.nvme0n1.out‚30257.41650390625( disk.nvme0c0n1.out‚30257.41650390625 disk.sda2.in‚0* proc.memory.percent‚16.920995712280273 cpu‚25.345706426515175) disk./.usagePercent‚17.44137140737459 disk.dm-1.out‚ 401.33984375 disk.rbd1.out‚ 3.6796875 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.671875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.671875$ memory_percent‚93.14043521881104 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0$ disk./.usageGB‚259.8890380859375 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda3.out‚ 401.33984375 network.sent‚ 2.1379795e+07 network.recv‚ 2.596946e+06 disk.drbd2265.in‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1123.87109375 disk.drbd2080.in‚0# disk.dm-0.out‚30257.41650390625 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3359375 proc.cpu.threads‚17ûë,½’Ô"‚(:‡ …ÜäÔ‘³£ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cb>gsÕ"‚(:è …ÜäÔÓŀ disk.drbd2080.out‚0 disk.sda.out‚ 401.71484375 disk.rbd0.out‚0 network.sent‚ 2.1488625e+07 disk.drbd2080.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 401.71484375 disk.sda1.in‚0 disk.nvme0n1.in‚4.671875% disk.nvme0n1.out‚30332.0927734375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.671875 disk.sda2.out‚0$ disk./.usageGB‚259.9100036621094 disk.drbd2265.in‚0$ memory_percent‚93.14041137695312* proc.memory.percent‚16.920995712280273 disk.rbd1.out‚ 3.6796875 disk.sda3.in‚ 126.77734375$ proc.memory.availableMB‚1123.875 disk.drbd2266.out‚0 network.recv‚ 2.603172e+06 disk.dm-0.in‚4.671875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 cpu‚25.345058142781344 disk.drbd1716.in‚0 disk.sda3.out‚ 401.71484375 disk.drbd2266.in‚0 disk.drbd2265.out‚0" disk.dm-0.out‚30332.0927734375 disk.sda1.out‚0' disk.nvme0c0n1.out‚30332.0927734375" proc.memory.rssMB‚ 2772.3359375* disk./.usagePercent‚17.4427784247827734==—’Ö"‚(:‡ ”ÜäÔ¦¯©~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\d­v×"‚(:ë ”ÜäÔ÷ꪀ disk.sda.out‚ 402.41796875 network.sent‚ 2.1498155e+07 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.671875 disk.dm-0.in‚4.671875" disk.dm-0.out‚30387.1357421875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚30387.1357421875 disk.drbd2265.out‚0 network.recv‚ 2.614701e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0$ disk./.usageGB‚259.9241600036621 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 cpu‚25.344447210932568 disk.sda1.in‚0% disk.nvme0n1.out‚30387.1357421875 disk.sda2.out‚0# memory_percent‚93.1420087814331 disk.sda3.out‚ 402.41796875) disk./.usagePercent‚17.44372846874234 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 402.41796875 disk.rbd1.out‚ 3.6796875 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.671875) proc.memory.availableMB‚ 1123.61328125" proc.memory.rssMB‚ 2772.3359375* proc.memory.percent‚16.920995712280273¥X562Ø"- _wandbruntime‚7202  _runtime‚72024§åÜÉÙ"à ' lr-AdamW/pg1‚3.1416831627207445e-05 ' lr-AdamW/pg2‚3.1416831627207445e-05 trainer/global_step‚8999 % _timestamp‚1.7884237025573523e+09 _runtime‚7202.294975463 _step‚737½É«OÎÚ"È trainer/global_step‚8999 % _timestamp‚1.7884237025573523e+09 _runtime‚7202.294975463 _step‚737 ' lr-AdamW/pg1‚3.1416831627207445e-05 ' lr-AdamW/pg2‚3.1416831627207445e-05áÑ/¬|2Û"- _wandbruntime‚7202  _runtime‚7202åF ךÜ"áO _runtime‚7202.295775754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43074810 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300834557 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7538990285312 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571580492762624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571580468822528 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119460776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚509717423401984 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7511234940118 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 trainer/global_step‚8999 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7908056677376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7538333922816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130917042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚530975295104000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 157332480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523436304818688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523436286815744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7510580502362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517227986864986 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579488525499904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169937553 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169917515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76385966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917515 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43074374 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7908214009856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119460338 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854597 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884237025580573e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488706772480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7538990285312 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚579488525499904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517228658342102 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130917044 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169937553 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 169917515 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚76385966 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169917515 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43074374 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7908214009856 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119460338 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚530974620738560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300854597 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚509717406362624 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130917044 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7884237025580573e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚579488706772480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7538333922816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76385964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7538990285312 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7510580502362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517227986864986 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 181272576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844âM¶£2Þ"- _wandbruntime‚7202  _runtime‚7202¡g²ï&ß"éM P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153350656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571642119560192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531031793205248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523492639854592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7202.98763675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130931060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579551227154944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169956033 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531032467584000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7909083654656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119473475 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43079341 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579551041701376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7512051375290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531032467584000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8999 _step‚739 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130931058 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169956033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788423703249526e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119473034 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7539809726464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76394132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43078902 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¿ xƒ"à"îM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76394134 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130931060 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚523492639854592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7539809726464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76394132 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43078902 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚523492657857536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7539153350656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚571642119560192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚531031793205248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169935969 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚523492639854592 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7202.98763675 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130931060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579551227154944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169956033 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531032467584000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523492657857536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7909083654656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300887093 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7539153350656 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517283707430510 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119473475 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43079341 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚579551041701376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7512051375290 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531032467584000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8999 _step‚739 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300887093 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169935969 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130931058 SEDeviceStatsMonitor.on_train_batch_end/alUªJGqlocation.small_pool.allocated‚ 169956033 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 $ _timestamp‚1.788423703249526e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7539809726464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119473034 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7951088420352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170843567 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120095852 _runtime‚7238.954881822 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43301486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932852224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582614279592960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574663005705728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚742 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533833932852224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131617844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7951249947136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131617842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582614094126080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884237392170982e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279526 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0N¨ÊCì'í"æO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533833932852224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚582614279592960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚574663005705728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚742 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533833932852224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533833258486784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7551415841770 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131617844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚512462612890624 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7951249947136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131617842 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚582614094126080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884237392170982e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7552070279526 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚520014700209510 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170863605 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7579976091136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170863605 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7951088420352 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚520014028732394 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 170843567 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120095852 _runtime‚7238.954881822 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7579319728640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533833258486784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574663029645824 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302461409 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43301486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0æ[G%$2î"- _wandbruntime‚7239  _runtime‚7239h|pëï"êM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚526310309799936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520069749297918 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676596443136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533891105332224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170882085 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 trainer/global_step‚9049 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚526310309799936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520070420775242 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120108225 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676788188160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951939999744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882085 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884237399073358e+09 _step‚743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493879 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚574724680383488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 964369902TžiÑ5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7580139156480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76802532 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276734 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43305693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7952107804672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120108679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 7239.64531144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43306145 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139156480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0hn•$õ&ð"ïM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302513945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43306145 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76802534 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7580139156480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131631860 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚526310309799936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚520069749297918 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7552886714698 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302493879 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚582676596443136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚533891105332224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170882085 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 trainer/global_step‚9049 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131631858 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚533890430953472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚526310309799936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚454 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚520070420775242 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120108225 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚582676788188160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7951939999744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170882085 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526310291796992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚574724656443392 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚512517517021184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884237399073358e+09 _step‚743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302493879 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚574724680383488 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚512517534060544 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131631858 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7580139156480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76802532 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7552232276734 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170862021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43305693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7952107804672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚526310291796992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120108679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 337538560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚ 7239.64531144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120733301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536692570600448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529071590700544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789657 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚577745566529024 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7994193456128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305534464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77202764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305534464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570600448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304108301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7994193456128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7620305534464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77202764 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7592251181178 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚536691896235008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7620305534464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚536692570600448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 304108301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚577745542588928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522800070599802 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120733301 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚536692570600448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529071608703488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529071590700544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7592905618934 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 171789657 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132318642 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0êòA–R2€#- _wandbruntime‚7276  _runtime‚7276 ôð&#êM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7593722054106 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781338112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7621124962304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120745528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7995218767360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7995055156736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884237768800414e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171808137 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585802248483328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43534596 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585802436034048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577807217266688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171788073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068701696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚747 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515262740588544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77210932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7621124962304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120746002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43535068 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529127961742336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621781338112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚536749743080448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7621124962304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304140797 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9099 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304120731 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120745528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7995218767360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132332658 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7995055156736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884237768800414e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 171808137 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚585802248483328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529127961742336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132332660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43534596 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚585802436034048 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚577807217266688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚536749743080448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚577807193326592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171788073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚536749068701696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚747 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 171808137 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚515262740588544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77210932 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7621124962304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132332658 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529127943739392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120746002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43535068 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77210934 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132332660 EYÜ’ˆ#‚(:‡ ýÜäÔäý} gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S›T£t‰#‚(:é ýÜäÔþ´ô disk.dm-0.in‚4.6875* disk./.usagePercent‚17.448498913135374 disk.sda.out‚ 406.04296875 disk.drbd1716.in‚0$ memory_percent‚93.14372539520264) proc.memory.availableMB‚ 1123.33203125 network.recv‚ 2.650519e+06 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 406.04296875 cpu‚25.34013074352628% disk./.usageGB‚259.99524307250977 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 406.04296875 disk.rbd1.out‚ 3.6796875 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0( disk.nvme0c0n1.out‚30810.11865234375 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.920995712280273 proc.cpu.threads‚17 disk.drbd2080.out‚0# disk.dm-0.out‚30810.11865234375 disk.sda2.in‚0" proc.memory.rssMB‚ 2772.3359375 network.sent‚ 2.1847646e+07& disk.nvme0n1.out‚30810.11865234375 disk.rbd0.in‚0 disk.sda2.out‚0|÷½É2Š#- _wandbruntime‚7312  _runtime‚7312£¢É‹#à ' lr-AdamW/pg2‚2.8623273261404925e-05 trainer/global_step‚9149 % _timestamp‚1.7884238126821315e+09 _runtime‚7312.419641659 _step‚749 ' lr-AdamW/pg1‚2.8623273261404925e-05å> ÎŒ#È trainer/global_step‚9149 % _timestamp‚1.7884238126821315e+09 _runtime‚7312.419641659 _step‚749 ' lr-AdamW/pg1‚2.8623273261404925e-05 ' lr-AdamW/pg2‚2.8623273261404925e-05í‡6×Ó2#- _wandbruntime‚7312  _runtime‚7312V¢Væ'Ž#àO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121368943 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517953025946624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588865375035392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8037295563264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7661291340288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525586112467210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77611166 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944326 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172715709 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 $ _timestamp‚1.788423812682835e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539551208348672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588865558405120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7633740958342 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8037454992896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9149 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208348672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43758223 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÙÌÒ[ë'#åO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121368943 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚517953025946624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚588865375035392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8037295563264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77611164 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7661291340288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525586112467210 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77611166 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525586783944326 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172715709 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 $ _timestamp‚1.788423812682835e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚539551208348672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚588865558405120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7633740958342 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚531889242642944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚531889260645888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133019444 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚531889242642944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚750 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚539550533983232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 172695671 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚531889260645888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8037454992896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7633086520586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚9149 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚539551208348672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133019442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43758223 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚580828079472128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚539550533983232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172715709 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0î¨D'2#- _wandbruntime‚7313  _runtime‚7313”›·¬[‘#êM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7662110768128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121381691 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580889754149888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133033458 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531945613684736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8038321752064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43762788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7313.341449938 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032734 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172714125 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380828672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588928075901952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788423813603384e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927894642688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767143936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110768128 H:DeviceStatsMonitor.on_tra¯žË=• in_batch_end/active.all.allocated‚ 305767649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121382122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43762359 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393514 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518007947116544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305767649 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133033458 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚531945613684736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8038321752064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43762788 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7313.341449938 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305747583 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚531945595681792 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚525641833032734 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7633902955550 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133033460 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172714125 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133033460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133033458 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚539608380828672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚588928075901952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 $ _timestamp‚1.788423813603384e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172734189 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7662767143936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚431 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚429 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚588927894642688 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚580889730209792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7662767143936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7662110768128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305767649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121382122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43762359 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7634557393514 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚531945595681792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518007947116544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚539608380828672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121381691 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚525642504510058 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77619334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580889754149888ïÜbÝm2“#- _wandbruntime‚7313  _runtime‚7313_qÙV”#Ð 1loss/SRqqt1/detection_loss‚1.8602915398384396 .loss/FBt1/detection_loss‚4.397755642731984 .loss/FBt2/detection_loss‚2.645794177055359 /loss/FBt4/detection_loss‚0.5533178758311581 .loss/FRt2/assignment_loss‚3.46240063394819 .loss/FRt3/detection_loss‚2.640372558683157 1loss/FBt1/assignment_loss‚0.38360045353571576 -loss/FRt2/detection_loss‚2.68052784545081 epoch‚7 % _timestamp‚1.7884238136050436e+09 _step‚752 2loss/SRqqt2/assignment_loss‚1.6288876845723106 2loss/SRqqt3/assignment_loss‚1.2542032642370982 1loss/FBt4/assignment_loss‚0.13035992076450423 .loss/FRt4/detection_loss‚1.435394114057999 1loss/SRqqt3/detection_loss‚1.8384104869935824 trainer/global_step‚9149 0loss/FRt3/assignment_loss‚2.6406582623720167 0loss/FRt1/assignment_loss‚3.6406111688148686 2loss/SRqqt1/assignment_loss‚0.8809104099085456 0loss/SRqqt2/detection_loss‚3.063083066826775 %loss/total_loss‚43.80758455972897 /loss/FRt4/assignment_loss‚1.122387525625527 2loss/SRqqt4/assignment_loss‚0.9921570581606995 1loss/SRqqt4/detection_loss‚0.8474503414521755 _runtime‚7313.342499809 1loss/FBt2/assignment_loss‚0.15908996704953324 .loss/FRt1/detection_loss‚2.124349645725111 .loss/FBt3/detection_loss‚3.089046669006348 0loss/FBt3/assignment_loss‚0.3365242470870612V2~[•#Õ .loss/FRt2/assignment_loss‚3.46240063394819 .loss/FRt3/detection_loss‚2.640372558683157 1loss/FBt1/assignment_loss‚0.38360045353571576 -loss/FRt2/detection_loss‚2.68052784545081 epoch‚7 % _timestamp‚1.7884238136050436e+09 _step‚752 2loss/SRqqt2/assignment_loss‚1.6288876845723106 2loss/SRqqt3/assignment_loss‚1.2542032642370982 1loss/FBt4/assignment_loss‚0.13035992076450423 .loss/FRt4/detection_loss‚1.435394114057999 1loss/SRqqt3/detection_loss‚1.8384104869935824 trainer/global_step‚9149 0loss/FRt3/assignment_loss‚2.6406582623720167 0loss/FRt1/assignment_loss‚3.6406111688148686 2loss/SRqqt1/assignment_loss‚0.8809104099085456 0loss/SRqqt2/detection_loss‚3.063083066826775 %loss/total_loss‚43.80758455972897 /loss/FRt4/assignment_loss‚1.122387525625527 2loss/SRqqt4/assignment_loss‚0.9921570581606995 1loss/SRqqt4/detection_loss‚0.8474503414521755 _runtime‚7313.342499809 1loss/FBt2/assignment_loss‚0.15908996704953324 .loss/FRt1/detection_loss‚2.124349645725111 .loss/FBt3/detection_loss‚3.089046669006348 0loss/FBt3/assignment_loss‚0.3365242470870612 1loss/SRqqt1/detection_loss‚1.8602915398384396 .loss/FBt1/detection_loss‚4.397755642731984 .loss/FBt2/detection_loss‚2.645794177055359 /loss/FBt4/detection_loss‚0.5533178758311581ðEWEó’–#‚(:‡ ŒÝäÔ¥·ª~ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìi7_—#‚(:Ô ŒÝäÔÍûê network.sent‚ 2.1853856e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 406.69921875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 30863.9375" proc.memory.rssMB‚ 2772.3359375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 cpu‚25.33955859477593 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 406.69921875 disk.sda1.in‚0 disk.sda.out‚ 406.69921875* proc.memory.percent‚16.920995712280273 network.recv‚ 2.654895e+06* disk./.usagePercent‚17.447685321191557 disk.nvme0n1.out‚ 30863.9375! disk.nvme0c0n1.out‚ 30863.9375 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0$ disk./.usageGB‚259.9831199645996 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 3.6796875 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875$ memory_percent‚93.14391613006592) proc.memory.availableMB‚ 1123.30078125°Ù±á‘˜#‚(:† ›ÝäÔ廲~ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.055' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.411 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šPt™#‚(:é ›ÝäÔâÀ€ disk.sda.out‚ 407.12109375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2772.3359375 disk.drbd2265.in‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 1123.296875 network.sent‚ 2.1961639e+07 network.recv‚ 2.659889e+06 disk.sda2.in‚0* proc.memory.percent‚16.920995712280273 disk.sda3.out‚ 407.12109375 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0$ memory_percent‚93.14393997192383 disk.sda3.in‚ 126.77734375 disk.sda1.in‚0& disk.nvme0n1.out‚30933.44873046875 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.448915437393584% disk./.usageGB‚260.00144958496094# disk.dm-0.out‚30933.44873046875 disk.drbd2266.in‚0 cpu‚25.338944070871786 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚30933.44873046875 disk.dm-1.out‚ 407.12109375 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.6796875 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375A–2š#- _wandbruntime‚7348  _runtime‚7348³Å|*Ç›#Á & lr-AdamW/pg2‚2.771505469926929e-05 trainer/global_step‚9199 % _timestamp‚1.7884238492457933e+09 _runtime‚7348.983320607 _step‚753 & lr-AdamW/pg1‚2.771505469926929e-05+aÌÌœ#Æ & lr-AdamW/pg1‚2.771505469926929e-05 & lr-AdamW/pg2‚2.771505469926929e-05 trainer/global_step‚9199 % _timestamp‚1.7884238492457933e+09 _runtime‚7348.983320607 _step‚753ñºÐü 2#- _wandbruntime‚7348  _runtime‚7348‹‚?~æ'ž#àO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534706894585344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8080302125568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277146112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7348.984331785 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43986351 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173641761 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334618 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43986789 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173621723 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8080463652352 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277146112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚754 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621723 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122005915 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846096896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811734 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122006355 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133720242 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133720244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43986789 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78019564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚542409171731456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 173621723 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8080463652352 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚534706912588288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7702277146112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚754 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 6714Otü·”77116 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307362005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78019566 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 173621723 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122005915 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚542409846096896 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚528372825811734 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122006355 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚542409171731456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚534706894585344 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307341965 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7702277146112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚7348.984331785 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7702933508608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43986351 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 173641761 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚583910616355328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚528372154334618ò.Ma2 #- _wandbruntime‚7349  _runtime‚73498¯¾Qð&¡#êM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542467018576896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534763265627136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7703752949760 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133734258 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520753136605184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚534763265627136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚528427874900142 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307394501 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚542466344198144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚9199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307374435 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173660241 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173640177 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592053431965696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8081164872704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122019017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294958 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122018578 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528428546377466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788423849938953e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43990846 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592053431965696 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307394501 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8081164872704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133734258 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚534763247624192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122019017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚755 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7674738294958 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122018578 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚528428546377466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 $ _timestamp‚1.788423849938953e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133734260 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43990846 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78027734 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7675392732922 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7743919314432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122642974 _step‚758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7714757199402 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8123292094976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚537524546527744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884238861869304e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174567813 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174547775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586993153238528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122643421 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545268483845120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116445333504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483845120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116624508928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7715411637158 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44215455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7385.924686997 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174567813èúËì'±#æO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚447 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122642974 _step‚758 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174547775 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7714757199402 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308988857 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚523443439002624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545267809479680 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134421044 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8123292094976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9249 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7743919314432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586993177178624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚537524546527744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134421042 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78427964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚523443456041984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 % _timestamp‚1.7884238861869304e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308968817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 174567813 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174547775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78427966 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚586993153238528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 122643421 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚537524564530688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531158867679142 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚545268483845120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545267809479680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595116445333504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545268483845120 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚595116624508928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7715411637158 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7743262951936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7743262951936 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44215455 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚7385.924686997 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174567813 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚537524564530688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531158196202026 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134421044 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7743919314432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952öEæ82²#- _wandbruntime‚7386  _runtime‚7386n›Bð&³#êM K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586293 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 7386.62474024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚523498360172544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚537580899566592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545325656325120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78436132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595178950736896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7744738755584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738755584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884238868864765e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122655668 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122656118 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44219536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595179140384768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537580899566592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7744082379776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚9249 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44219984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8124146760704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚02“Úê´#ïM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 7386.62474024 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allW ™ %ocated‚523498360172544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚537580899566592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545324981946368 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚545325656325120 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78436132 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595178950736896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7744738755584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚587054827916288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309001287 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7744738755584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531213916767550 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7715573634366 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174586293 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚587054803976192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884238868864765e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122655668 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚523498343133184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚537580917569536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122656118 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309001287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44219536 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309021353 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595179140384768 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134435060 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚537580899566592 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7744082379776 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7744082379776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚759 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 trainer/global_step‚9249 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134435058 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78436134 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44219984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8124146760704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134435058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174586293 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175493865 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7755592538810 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7784248757760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8166213207040 _step‚762 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8166378928128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310615709 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175493865 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7756246976566 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚548127121593344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78836364 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135121844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952Y )F·Á#æO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135121844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚598242092989952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 175493865 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚540342216473088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7755592538810 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚548126447227904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚548126447227904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7784248757760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8166213207040 _step‚762 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175473827 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8166378928128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310615709 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.pea¤G¼™5k‚ 339635712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175493865 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7756246976566 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚548127121593344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78836364 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135121844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚590075690121728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088ú°“W2Â#- _wandbruntime‚7422  _runtime‚7422£Øáð&Ã#êM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294073344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78844534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123292090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724561408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 % _timestamp‚1.7884239231722906e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175492281 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304415232000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8167235886080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚540398551508992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44447558 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8167074372608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598304600685568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44447991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7785724561408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512345 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7756408973774®SDõ&Ä#ïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚526243549661184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚548184294073344 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚548184294073344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78844534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590137364799488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 trainer/global_step‚9299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚526243566700544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123292090 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7785724561408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 % _timestamp‚1.7884239231722906e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚540398551508992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78844532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533999958634958 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7785068185600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175492281 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚598304415232000 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8167235886080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚540398551508992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚540398569511936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7785068185600 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135135860 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44447558 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8167074372608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚763 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚598304600685568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135135858 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310648205 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44447991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310628139 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚540398569511936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚435 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175492281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7785724561408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚534000630112282 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135135858 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175512345 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7756408973774 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135135860 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ûKÙùM2Å#- _wandbruntime‚7422  _runtime‚7422OIFfZÆ#Ô trainer/global_step‚9299 /loss/FRt3/assignment_loss‚2.370917794439528 2loss/SRqqt1/assignment_loss‚1.0466281648365443 .loss/FBt3/detection_loss‚2.863377398914761 _runtime‚7422.911344893 2loss/SRqqt2/assignment_loss‚1.7163959013727994 .loss/FRt4/detection_loss‚1.423745958941678 0loss/SRqqt2/detection_loss‚2.673970157137284 /loss/FRt2/detection_loss‚2.7849069819217775 0loss/SRqqt3/detection_loss‚1.726830663504424 % _timestamp‚1.7884239231739326e+09 &loss/total_loss‚41.278701173555675 0loss/FRt2/assignment_loss‚3.4818527291460737 2loss/SRqqt3/assignment_loss‚1.6382511616167095 2loss/SRqqt4/assignment_loss‚0.8604147486846175 0loss/FBt3/assignment_loss‚0.1997060360865564 /loss/FRt3/detection_loss‚2.3879576477739546 .loss/FBt2/detection_loss‚2.439179003238678 /loss/FRt1/assignment_loss‚3.029145233514832 0loss/FRt4/assignment_loss‚1.5791251003742217 1loss/FBt2/assignment_loss‚0.33227970333743834 /loss/FRt1/detection_loss‚1.9389209703701549 1loss/FBt4/assignment_loss‚0.19184474429637474 epoch‚7 1loss/SRqqt4/detection_loss‚1.1592440327400702 /loss/FBt1/detection_loss‚2.7207066317399344 0loss/FBt1/assignment_loss‚0.2769138183599959 0loss/SRqqt1/detection_loss‚1.900357249108228 /loss/FBt4/detection_loss‚0.5360293420990381 _step‚764°,1_Ç#Ù 0loss/FBt1/assignment_loss‚0.2769138183599959 0loss/SRqqt1/detection_loss‚1.900357249108228 /loss/FBt4/detection_loss‚0.5360293420990381 _step‚764 trainer/global_step‚9299 /loss/FRt3/assignment_loss‚2.370917794439528 2loss/SRqqt1/assignment_loss‚1.0466281648365443 .loss/FBt3/detection_loss‚2.863377398914761 _runtime‚7422.911344893 2loss/SRqqt2/assignment_loss‚1.7163959013727994 .loss/FRt4/detection_loss‚1.423745958941678 0loss/SRqqt2/detection_loss‚2.673970157137284 /loss/FRt2/detection_loss‚2.7849069819217775 0loss/SRqqt3/detection_loss‚1.726830663504424 % _timestamp‚1.7884239231739326e+09 &loss/total_loss‚41.278701173555675 0loss/FRt2/assignment_loss‚3.4818527291460737 2loss/SRqqt3/assignment_loss‚1.6382511616167095 2loss/SRqqt4/assignment_loss‚0.8604147486846175 0loss/FBt3/assignment_loss‚0.1997060360865564 /loss/FRt3/detection_loss‚2.3879576477739546 .loss/FBt2/detection_loss‚2.439179003238678 /loss/FRt1/assignment_loss‚3.029145233514832 0loss/FRt4/assignment_loss‚1.5791251003742217 1loss/FBt2/assignment_loss‚0.33227970333743834 /loss/FRt1/detection_loss‚1.9389209703701549 1loss/FBt4/assignment_loss‚0.19184474429637474 epoch‚7 1loss/SRqqt4/detection_loss‚1.1592440327400702 /loss/FBt1/detection_loss‚2.7207066317399344ü*¡_‡È#‚(:… õÝäÔ§ƒ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T}ÐËpÉ#‚(:å õÝäÔÎÿ disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚31292.59423828125 disk.rbd1.out‚3.828125 disk.sda.in‚ 126.77734375 disk.sda.out‚ 410.9765625) proc.memory.availableMB‚ 1123.18359375" proc.memory.rssMB‚ 2772.3359375 cpu‚25.335390512759183 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚31292.59423828125 disk.sda3.out‚ 410.9765625 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 410.9765625 disk.drbd2266.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.920995712280273 disk.sda2.in‚0 disk.nvme0n1.in‚4.6875 network.sent‚ 2.2216625e+07* disk./.usagePercent‚17.454857116255766 disk.drbd1716.in‚0& disk.nvme0n1.out‚31292.59423828125 disk.sda1.in‚0$ disk./.usageGB‚260.0899848937988 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 2.716455e+06 disk.rbd0.out‚0$ memory_percent‚93.14463138580322V€t ’Ê#‚(:‡ „ÞäÔ†” ~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘…Å‹jË#‚(:ß „ÞäÔ±À®€ disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 411.671875 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚4.6875 network.sent‚ 2.2332397e+07 disk.sda1.out‚0" proc.memory.rssMB‚ 2772.3359375 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.out‚ 411.671875 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875& disk.nvme0n1.out‚31350.96142578125 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚31350.96142578125$ memory_percent‚93.14775466918945 disk.rbd1.out‚3.828125 disk.sda1.in‚0' proc.memory.availableMB‚ 1122.671875 cpu‚25.33481452566709 disk.drbd1716.out‚0 network.recv‚ 2.736065e+06# disk./.usageGB‚260.106689453125 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 411.671875 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.920995712280273 disk.rbd1.in‚0* disk./.usagePercent‚17.455978173248205# disk.dm-0.out‚31350.96142578125 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0KHXÌ#‚(:’ “ÞäÔÌǺ~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Kø¥rÍ#‚(:ç “ÞäÔ¦ó¸€ disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 412.234375" proc.memory.rssMB‚ 2772.3359375 network.recv‚ 2.739075e+06 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 cpu‚25.334242893076258 disk.sda1.out‚0 disk.drbd2080.out‚0$ disk./.usageGB‚260.0808410644531 disk.dm-0.in‚4.6875# disk.dm-0.out‚31399.88720703125 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0& disk.nvme0n1.out‚31399.88720703125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.rbd0.in‚0 network.sent‚ 2.2336814e+07* disk./.usagePercent‚17.454243466196193 disk.sda3.out‚ 412.234375 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1123.16015625* proc.memory.percent‚16.920995712280273 disk.rbd1.out‚ 3.8603515625 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚31399.88720703125 disk.drbd2265.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.6875 disk.dm-1.out‚ 412.234375 disk.sda1.in‚0$ memory_percent‚93.14477443695068žšá72Î#- _wandbruntime‚7458  _runtime‚7458Ø*ÉÏ#à ' lr-AdamW/pg2‚2.5061951215578292e-05 trainer/global_step‚9349 % _timestamp‚1.7884239592310553e+09 _runtime‚7458.968609727 _step‚765 ' lr-AdamW/pg1‚2.5061951215578292e-05±1øÎÐ#È ' lr-AdamW/pg1‚2.5061951215578292e-05 ' lr-AdamW/pg2‚2.5061951215578292e-05 trainer/global_step‚9349 % _timestamp‚1.7884239592310553e+09 _runtime‚7458.968609727 _step‚765ýêÓ2Ñ#- _wandbruntime‚7458  _runtime‚7458püàEÒ#áO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7458.969649338 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176399879 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315974 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADevice¢a±¢#StatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550985759341568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543159868415488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419917 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44672227 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44671778 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚9349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7825234563584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536730951413958 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123916542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884239592318935e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135822644 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399879 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601367561506304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8209334501376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8209496028160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543159850412544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550985759341568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123916993 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7458.969649338 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176399879 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7797082315974 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312222521 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135822644 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312242561 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚550985759341568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543159868415488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7825890926080 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176419917 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44672227 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚550985084976128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135822642 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79244766 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44671778 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 trainer/global_step‚9349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7825234563584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚550985084976128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚536730951413958 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543159850412544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123916542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593158250945024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 % _timestamp‚1.7884239592318935e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135822644 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543159868415488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 176399879 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601367561506304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8209334501376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8209496028160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543159850412544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚550985759341568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135822642 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚766 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123916993 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543216203451392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601430262880768 trainer/global_step‚9349 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176438397 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123929245 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚536786000502366 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312254991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751146 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593219901682688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543216203451392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176438397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8210361198080 $ _timestamp‚1.788423959914469e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135836660 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7459.652387775 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135836660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44676313 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123929702 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430zãã- Ö#ïM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593219901682688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚543216203451392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312275057 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135836658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593219877742592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7826053991424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176438397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7797244313182 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8210361198080 $ _timestamp‚1.788423959914469e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135836660 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79252934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _runtime‚7459.652387775 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528988756189184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135836660 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44676313 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 179162112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123929702 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDevicÑZ]ÈeStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚551042931821568 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _step‚767 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312254991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚543216221454336 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543216203451392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601430262880768 trainer/global_step‚9349 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312275057 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176438397 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7826710367232 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123929245 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚536786000502366 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312254991 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚457 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7826710367232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚551042931821568 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135836658 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚551042257442816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7797898751146ÿ>“2×#- _wandbruntime‚7459  _runtime‚7459mà2WØ#Ñ 0loss/FBt1/assignment_loss‚0.2834787270985544 /loss/FRt4/detection_loss‚1.7565927726527055 %loss/total_loss‚44.34361970807638 2loss/FBt3/assignment_loss‚0.020989330601878464 .loss/FRt3/detection_loss‚2.276696264181497 1loss/SRqqt1/detection_loss‚1.8079107018078076 /loss/FBt1/detection_loss‚2.3806550204753876 .loss/FBt3/detection_loss‚4.403975874185562 epoch‚7 /loss/FRt3/assignment_loss‚2.604896315988505 1loss/SRqqt3/assignment_loss‚1.233669341162399 2loss/SRqqt4/assignment_loss‚1.0174758500144097 1loss/SRqqt3/detection_loss‚1.9063602812864162 .loss/FBt4/assignment_loss‚0.12306364076802 /loss/FBt4/detection_loss‚0.5254728387305398 $ _timestamp‚1.788423959915993e+09 /loss/FRt2/detection_loss‚2.8234521185650543 0loss/FBt2/assignment_loss‚0.5070432580076158 .loss/FRt1/detection_loss‚2.433559048743475 _step‚768 /loss/FRt4/assignment_loss‚1.480253235499064 2loss/SRqqt1/assignment_loss‚1.5718993873280638 2loss/SRqqt2/assignment_loss‚2.0593602278015832 .loss/FBt2/detection_loss‚2.855527341365814 trainer/global_step‚9349 _runtime‚7459.653410855 1loss/SRqqt2/detection_loss‚2.4566420804370535 1loss/SRqqt4/detection_loss‚0.9622078254084352 0loss/FRt1/assignment_loss‚3.0913341471127103 0loss/FRt2/assignment_loss‚3.7611040788538315¦³T\Ù#Ö 1loss/SRqqt2/detection_loss‚2.4566420804370535 1loss/SRqqt4/detection_loss‚0.9622078254084352 0loss/FRt1/assignment_loss‚3.0913341471127103 0loss/FRt2/assignment_loss‚3.7611040788538315 0loss/FBt1/assignment_loss‚0.2834787270985544 /loss/FRt4/detection_loss‚1.7565927726527055 %loss/total_loss‚44.34361970807638 2loss/FBt3/assignment_loss‚0.020989330601878464 .loss/FRt3/detection_loss‚2.276696264181497 1loss/SRqqt1/detection_loss‚1.8079107018078076 /loss/FBt1/detection_loss‚2.3806550204753876 .loss/FBt3/detection_loss‚4.403975874185562 epoch‚7 /loss/FRt3/assignment_loss‚2.604896315988505 1loss/SRqqt3/assignment_loss‚1.233669341162399 2loss/SRqqt4/assignment_loss‚1.0174758500144097 1loss/SRqqt3/detection_loss‚1.9063602812864162 .loss/FBt4/assignment_loss‚0.12306364076802 /loss/FBt4/detection_loss‚0.5254728387305398 $ _timestamp‚1.788423959915993e+09 /loss/FRt2/detection_loss‚2.8234521185650543 0loss/FBt2/assignment_loss‚0.5070432580076158 .loss/FRt1/detection_loss‚2.433559048743475 _step‚768 /loss/FRt4/assignment_loss‚1.480253235499064 2loss/SRqqt1/assignment_loss‚1.5718993873280638 2loss/SRqqt2/assignment_loss‚2.0593602278015832 .loss/FBt2/detection_loss‚2.855527341365814 trainer/global_step‚9349 _runtime‚7459.653410855€©Âþ¢‘Ú#‚(:† ¢ÞäÔõ÷ˆ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.805' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.761 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cκ„sÛ#‚(:è ¢ÞäÔΤ‚ disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 412.68359375 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.8603515625 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚31447.64501953125& proc.memory.availableMB‚ 1123.09375 proc.cpu.threads‚17$ disk./.usageGB‚260.0946235656738 disk.drbd2266.out‚0& disk.nvme0n1.out‚31447.64501953125 disk.rbd0.out‚0( disk./.usagePercent‚17.4551684214174 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375 disk.sda1.in‚0" proc.memory.rssMB‚ 2772.3359375 network.recv‚ 2.743679e+06 disk.drbd2080.out‚0 disk.sda.out‚ 412.68359375 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.920995712280273 cpu‚25.333670147372153 network.sent‚ 2.2342839e+07 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875$ memory_percent‚93.14517974853516# disk.dm-0.out‚31447.64501953125 disk.drbd2266.in‚0 disk.sda3.out‚ 412.68359375óñC’Ü#‚(:‡ ±ÞäÔøñ€~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä{5oÝ#‚(:ä ±ÞäÔ„Ýþ€ disk.sda2.out‚0! disk.dm-0.out‚31517.314453125 disk.dm-1.out‚ 413.0859375" proc.memory.rssMB‚ 2772.3359375 cpu‚25.33310384742581 network.recv‚ 2.749805e+06 disk.rbd1.out‚ 3.92333984375 disk.sda.out‚ 413.0859375$ disk.nvme0n1.out‚31517.314453125 disk.sda2.in‚0 disk.sda3.out‚ 413.0859375) proc.memory.availableMB‚ 1123.33203125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 2.2451344e+07 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.drbd2266.in‚0$ memory_percent‚93.14372539520264* disk./.usagePercent‚17.454954911133846 disk.sda.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.920995712280273 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875& disk.nvme0c0n1.out‚31517.314453125 disk.dm-0.in‚4.6875 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0$ disk./.usageGB‚260.09144210815432~ü2Þ#- _wandbruntime‚7495  _runtime‚7495ÆMÞÇÈß# trainer/global_step‚9399 $ _timestamp‚1.788423995774703e+09 _runtime‚7495.512242566 _step‚769 ' lr-AdamW/pg1‚2.4202048462620466e-05 ' lr-AdamW/pg2‚2.4202048462620466e-05V¡°Íà#Ç $ _timestamp‚1.788423995774703e+09 _runtime‚7495.512242566 _step‚769 ' lr-AdamW/pg1‚2.4202048462620466e-05 ' lr-AdamW/pg2‚2.4202048462620466e-05 trainer/global_step‚9399°<52á#- _wandbruntime‚7495  _runtime‚7495œÝèPç'â#áO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325931 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281366 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545977502354944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7866876731904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596240787828224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313869413 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚7495.513147033 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604493190134784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553844397089792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8252426246656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531679058586624 trainer/global_step‚9399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44898651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596240763888128bƒ %ã#æO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚531679075625984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚770 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136523444 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7866876731904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539516321804250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177325931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7866220369408 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553843722724352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545977520357888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8252426246656 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚531679058586624 trainer/global_step‚9399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44898651 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚596240763888128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136523444 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 177325931 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539516993281366 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545977502354944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7866876731904 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177345969 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚596240787828224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553843722724352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313869413 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚7495.513147033 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313849373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚604493190134784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 177345969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7837917655382 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚553844397089792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553900895191040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177364449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553901569569792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136537460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44903197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8253277714432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124565001 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253447616512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚9399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7867039797248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚771 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124564529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136537460 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚7496.204806389 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881843 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344385 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039797248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555886182400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136537458 % _timestamp‚1.7884239964667895e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847098™t`+ö&æ#ðM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚604555692340224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚596302414625792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539572042369774 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚553900895191040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177364449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 313901909 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚546033873396736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚546033855393792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7867696173056 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚596302438565888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚546033855393792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚553901569569792 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136537460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79661334 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚531733962717184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177344385 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44903197 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8253277714432 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚531733979756544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124565001 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8253447616512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚9399 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7867039797248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚771 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79661332 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚553901569569792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124564529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136537460 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _runtime‚7496.204806389 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313881843 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177344385 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7867039797248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313901909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7838734090554 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313881843 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚604555886182400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚546033873396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136537458 % _timestamp‚1.7884239964667895e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539572713847098ƒ‰B®–2ç#- _wandbruntime‚7496  _runtime‚7496˜u|‰]è#× &loss/total_loss‚41.497061107164754 2loss/FBt1/assignment_loss‚0.060255901771597564 /loss/FBt2/detection_loss‚3.7657000571489334 _runtime‚ 7496.20616899 2loss/FBt2/assignment_loss‚0.021468646184075624 1loss/SRqqt2/detection_loss‚2.2863054968215324 .loss/FBt3/detection_loss‚2.469232416152954 0loss/FRt4/assignment_loss‚1.2247534605364006 1loss/SRqqt3/assignment_loss‚1.181344414457541 /loss/FRt1/detection_loss‚2.7120187862923273 .loss/FRt2/detection_loss‚2.729733605595196 1loss/SRqqt3/detection_loss‚1.6606884226202965 .loss/FBt4/detection_loss‚0.640375178841943 epoch‚7 trainer/global_step‚9399 /loss/FRt2/assignment_loss‚3.244597540182226 2loss/SRqqt1/assignment_loss‚1.3004374006798245 /loss/FRt4/detection_loss‚1.5918933860957623 /loss/FBt1/detection_loss‚2.5677808423837027 $ _timestamp‚1.788423996468431e+09 _step‚772 0loss/FRt1/assignment_loss‚3.0636954056589225 /loss/FRt3/assignment_loss‚2.911028946345707 2loss/SRqqt2/assignment_loss‚1.8988675773143768 1loss/FBt4/assignment_loss‚0.15370420524977346 1loss/SRqqt4/assignment_loss‚1.083922588530307 2loss/FBt3/assignment_loss‚0.029150105477310718 /loss/FRt3/detection_loss‚2.0188161352614187 1loss/SRqqt1/detection_loss‚1.7398629233522236 0loss/SRqqt4/detection_loss‚1.141427664210399ˆ¶KÄbé#Ü /loss/FRt3/assignment_loss‚2.911028946345707 2loss/SRqqt2/assignment_loss‚1.8988675773143768 1loss/FBt4/assignment_loss‚0.15370420524977346 1loss/SRqqt4/assignment_loss‚1.083922588530307 2loss/FBt3/assignment_loss‚0.029150105477310718 /loss/FRt3/detection_loss‚2.0188161352614187 1loss/SRqqt1/detection_loss‚1.7398629233522236 0loss/SRqqt4/detection_loss‚1.141427664210399 &loss/total_loss‚41.497061107164754 2loss/FBt1/assignment_loss‚0.060255901771597564 /loss/FBt2/detection_loss‚3.7657000571489334 _runtime‚ 7496.20616899 2loss/FBt2/assignment_loss‚0.021468646184075624 1loss/SRqqt2/detection_loss‚2.2863054968215324 .loss/FBt3/detection_loss‚2.469232416152954 0loss/FRt4/assignment_loss‚1.2247534605364006 1loss/SRqqt3/assignment_loss‚1.181344414457541 /loss/FRt1/detection_loss‚2.7120187862923273 .loss/FRt2/detection_loss‚2.729733605595196 1loss/SRqqt3/detection_loss‚1.6606884226202965 .loss/FBt4/detection_loss‚0.640375178841943 epoch‚7 trainer/global_step‚9399 /loss/FRt2/assignment_loss‚3.244597540182226 2loss/SRqqt1/assignment_loss‚1.3004374006798245 /loss/FRt4/detection_loss‚1.5918933860957623 /loss/FBt1/detection_loss‚2.5677808423837027 $ _timestamp‚1.788423996468431e+09 _step‚772 0loss/FRt1/assignment_loss‚3.0636954056589225„!ƒ¹t’ê#‚(:‡ ÀÞäÔÁà¦~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.206' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6412 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a'së#‚(:è ÀÞäÔᯯ€ disk.rbd1.out‚ 3.92333984375 disk.sda.in‚ 126.77734375$ memory_percent‚93.14546585083008 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875$ disk.nvme0n1.out‚31568.912109375! disk.dm-0.out‚31568.912109375 disk.sda.out‚ 413.71484375 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd2266.out‚0 network.recv‚ 2.761396e+06 disk.drbd2265.out‚0' proc.memory.availableMB‚ 1123.046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.6875& disk.nvme0c0n1.out‚31568.912109375 disk.dm-1.out‚ 413.71484375* proc.memory.percent‚16.921019554138184 cpu‚25.332510669025545 network.sent‚ 2.2460978e+07 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.dm-0.in‚4.6875* disk./.usagePercent‚17.456142274077653% disk./.usageGB‚260.10913467407227 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 413.71484375 disk.drbd2080.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2772.33984375 disk.drbd2265.in‚0 disk.drbd2266.in‚0\’‘ì#‚(:† ÏÞäÔ Ž—~ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÂäòmí#‚(:â ÏÞäÔ‡¼„ network.sent‚ 2.2569433e+07 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.921019554138184 cpu‚25.331977288426692 disk.nvme0n1.in‚4.6875# disk.nvme0n1.out‚31654.26171875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2772.33984375 proc.cpu.threads‚17$ disk./.usageGB‚260.1240997314453 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 414.23046875 disk.sda.out‚ 414.23046875 disk.dm-0.out‚31654.26171875$ memory_percent‚93.14546585083008 disk.sda2.out‚0* disk./.usagePercent‚17.457146591634476 disk.rbd1.out‚ 3.986328125 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.sda3.out‚ 414.23046875 disk.drbd2080.in‚0 disk.dm-0.in‚4.6875' proc.memory.availableMB‚ 1123.046875 network.recv‚ 2.767368e+06 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.6875% disk.nvme0c0n1.out‚31654.26171875 disk.drbd2265.in‚0¼¥î#‚(:’ ÞÞäÔÈß²~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.168' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.233599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;- Önï#‚(:ã ÞÞäÔáÁ—€ network.sent‚ 2.2575362e+07 network.recv‚ 2.771313e+06# disk.nvme0n1.out‚31712.91015625 cpu‚25.331423460020403 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda.out‚ 414.85546875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375) disk./.usagePercent‚17.45787365303172 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.986328125$ disk./.usageGB‚260.1349334716797 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0$ memory_percent‚93.14696788787842* proc.memory.percent‚16.921019554138184% disk.nvme0c0n1.out‚31712.91015625 disk.sda2.in‚0 disk.dm-0.in‚4.6875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1122.80078125 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.dm-0.out‚31712.91015625 disk.dm-1.out‚ 414.85546875 disk.sda3.out‚ 414.85546875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2772.339843759žÈp2ð#- _wandbruntime‚7532  _runtime‚7532ˆ\ÜÇñ#Á _step‚773 & lr-AdamW/pg1‚2.335467905704773e-05 & lr-AdamW/pg2‚2.335467905704773e-05 trainer/global_step‚9449 % _timestamp‚1.7884240327117345e+09 _runtime‚7532.449257153"ăÌò#Æ & lr-AdamW/pg1‚2.335467905704773e-05 & lr-AdamW/pg2‚2.335467905704773e-05 trainer/global_step‚9449 % _timestamp‚1.7884240327117345e+09 _runtime‚7532.449257153 _step‚773…MéQ:2ó#- _wandbruntime‚7532  _runtime‚7532a#! “ô#áO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206175232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125187305 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7532.450207095 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534424282153984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884240327124326e+09 _step‚774 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178251983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534424265114624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45125741 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548795172300288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178251983 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.alW=©ÕTlocated‚ 7878752994790 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45126185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607619187663872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206175232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556703034838016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542302363671658 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7907862537728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178272021 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137224244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80061566 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295693037056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125187751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599323300771328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8295862952448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862537728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9449 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315476225 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128ý½ßoì'õ#æO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7907206175232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125187305 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7532.450207095 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚556702360472576 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534424282153984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884240327124326e+09 _step‚774 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178251983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚542303035148774 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚534424265114624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137224242 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45125741 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚548795172300288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178251983 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚548795154297344 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7878752994790 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178272021 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45126185 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607619187663872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7907206175232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚556703034838016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚542302363671658 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7907862537728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178272021 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137224244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80061566 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295693037056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80061564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125187751 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚599323324711424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚599323300771328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7878098557034 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8295862952448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7907862537728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚9449 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315476225 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚548795172300288 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚556702360472576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚548795154297344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680†{š÷2ö#- _wandbruntime‚7533  _runtime‚7533‡=ð&÷#êM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207318016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7878914991998 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137238258 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315508695 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178270437 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚556759532939264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 315508695 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80069734 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137238260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607681517122560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚607681696284672 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚9449 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884240333954957e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548851525339136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125200046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681978880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207318016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178290501 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137238258 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429962 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45130314 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8296565613568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290501 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542358084237182 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 % _timestamp‚1.7884240333954957e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚548851525339136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125200046 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7908681978880 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚556760207318016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚542358755714506 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178290501 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534479169245184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534479186284544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚548851507336192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315528761 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137238258 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚599384975449088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7879569429962 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 178270437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45130314 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚599384951508992 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8296565613568 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178290501 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚548851525339136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚542358084237182 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315528761 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 155222016 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚556760207318016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7878914991998 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137238260 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2‡ä×L2ù#- _wandbruntime‚7533  _runtime‚7533ú®l®Wú#Ñ 1loss/SRqqt4/detection_loss‚0.9389019732479937 .loss/FBt2/detection_loss‚2.380036823451519 1loss/SRqqt1/assignment_loss‚1.159941504344992 2loss/SRqqt2/assignment_loss‚1.6878592874854803 .loss/FRt3/detection_loss‚2.214656636118889 % _timestamp‚1.7884240333972824e+09 0loss/FRt1/assignment_loss‚3.2779398020356894 0loss/FBt1/assignment_loss‚0.6292794285850092 1loss/FBt4/assignment_loss‚0.14244380247817884 0loss/SRqqt2/detection_loss‚2.072257698327303 0loss/SRqqt3/detection_loss‚1.923113768051068 /loss/FBt4/detection_loss‚0.5721985057055907 epoch‚7 /loss/FRt4/assignment_loss‚1.355826308992174 1loss/SRqqt4/assignment_loss‚0.716070790309459 0loss/FBt2/assignment_loss‚0.7014497376439977 /loss/FRt3/assignment_loss‚2.726754412651062 1loss/FBt3/assignment_loss‚0.13330742159159853 %loss/total_loss‚40.11393643513949 _runtime‚7533.134712712 0loss/FRt2/assignment_loss‚3.1003697637249443 2loss/SRqqt3/assignment_loss‚0.9584027989870972 1loss/SRqqt1/detection_loss‚1.8541290944685107 trainer/global_step‚9449 .loss/FRt1/detection_loss‚2.332217295654118 /loss/FRt2/detection_loss‚2.8267999449196983 .loss/FRt4/detection_loss‚1.568667767610815 /loss/FBt1/detection_loss‚2.1452764001759617 /loss/FBt3/detection_loss‚2.6960354685783385 _step‚776ÑjÆ\û#Ö 0loss/FRt2/assignment_loss‚3.1003697637249443 2loss/SRqqt3/assignment_loss‚0.9584027989870972 1loss/SRqqt1/detection_loss‚1.8541290944685107 trainer/global_step‚9449 .loss/FRt1/detection_loss‚2.332217295654118 /loss/FRt2/detection_loss‚2.8267999449196983 .loss/FRt4/detection_loss‚1.568667767610815 /loss/FBt1/detection_loss‚2.1452764001759617 /loss/FBt3/detection_loss‚2.6960354685783385 _step‚776 1loss/SRqqt4/detection_loss‚0.9389019732479937 .loss/FBt2/detection_loss‚2.380036823451519 1loss/SRqqt1/assignment_loss‚1.159941504344992 2loss/SRqqt2/assignment_loss‚1.6878592874854803 .loss/FRt3/detection_loss‚2.214656636118889 % _timestamp‚1.7884240333972824e+09 0loss/FRt1/assignment_loss‚3.2779398020356894 0loss/FBt1/assignment_loss‚0.6292794285850092 1loss/FBt4/assignment_loss‚0.14244380247817884 0loss/SRqqt2/detection_loss‚2.072257698327303 0loss/SRqqt3/detection_loss‚1.923113768051068 /loss/FBt4/detection_loss‚0.5721985057055907 epoch‚7 /loss/FRt4/assignment_loss‚1.355826308992174 1loss/SRqqt4/assignment_loss‚0.716070790309459 0loss/FBt2/assignment_loss‚0.7014497376439977 /loss/FRt3/assignment_loss‚2.726754412651062 1loss/FBt3/assignment_loss‚0.13330742159159853 %loss/total_loss‚40.11393643513949 _runtime‚7533.134712712ˆ¢–K¾’ü#‚(:‡ íÞäÔд~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.7902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Íïmý#‚(:â íÞäÔá¡° disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0) disk./.usagePercent‚17.45869287714129 disk.sda2.out‚0 disk.drbd2265.in‚0 network.recv‚ 2.775298e+06$ disk.nvme0n1.out‚31787.654296875 disk.drbd2080.in‚0 disk.dm-0.in‚4.6875) proc.memory.availableMB‚ 1122.98046875 disk.drbd1716.in‚0 cpu‚25.33085757194215 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-1.out‚ 415.3046875 disk.rbd1.in‚0 disk.sda.out‚ 415.3046875 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875! disk.dm-0.out‚31787.654296875# proc.memory.rssMB‚ 2772.33984375 disk.rbd1.out‚ 3.986328125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚31787.654296875 disk.sda3.out‚ 415.3046875* proc.memory.percent‚16.921019554138184 network.sent‚ 2.2581351e+07 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚93.14587116241455$ disk./.usageGB‚260.1471405029297 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375¡Î4 þ#‚(:’ üÞäÔÌÞ~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôqÿ#‚(:æ üÞäÔ‹€ disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 415.83984375 disk.dm-0.in‚4.6875 disk.drbd1716.in‚0$ memory_percent‚93.14587116241455 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.6875) proc.memory.availableMB‚ 1122.98046875* proc.memory.percent‚16.921019554138184 network.recv‚ 2.781831e+06) disk./.usagePercent‚17.45951210125086 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 415.83984375 disk.drbd1716.out‚0$ disk.nvme0n1.out‚31836.291015625& disk.nvme0c0n1.out‚31836.291015625 disk.drbd2080.in‚0 cpu‚25.330310955243547 proc.cpu.threads‚17# proc.memory.rssMB‚ 2772.33984375 network.sent‚ 2.2690287e+07$ disk./.usageGB‚260.1593475341797 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 415.83984375 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 3.986328125! disk.dm-0.out‚31836.291015625L0ôè2€$- _wandbruntime‚7569  _runtime‚7569‹¹ÞûÉ$à % _timestamp‚1.7884240694780161e+09 _runtime‚7569.215547248 _step‚777 ' lr-AdamW/pg1‚2.2520019589374147e-05 ' lr-AdamW/pg2‚2.2520019589374147e-05 trainer/global_step‚9499¦`΂$È _step‚777 ' lr-AdamW/pg1‚2.2520019589374147e-05 ' lr-AdamW/pg2‚2.2520019589374147e-05 trainer/global_step‚9499 % _timestamp‚1.7884240694780161e+09 _runtime‚7569.215547248‰8Gm¢2ƒ$- _wandbruntime‚7569  _runtime‚75690wç'„$áO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744709111808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317123117 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744905064448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45354820 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178035 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672586240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7948848343552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125824786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 % _timestamp‚1.7884240694787343e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8339043469824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚778 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602405861594624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338871457280üþñ¤ì'…$æO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80469964 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7948848343552 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125824786 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179178035 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚545089077016182 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7948191981056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚602405837654528 % _timestamp‚1.7884240694787343e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8339043469824 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7948191981056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚778 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559560998220800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚551612806239744 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80469966 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚602405861594624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8338871457280 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚551612806239744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610744709111808 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 179198073 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 317123117 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610744905064448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚559560998220800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45354820 trainer/global_step‚9499 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179178035 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559561672586240 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚551612824242688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317103077 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7948848343552 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚537169488681984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137925042 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125837056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7920404769370 >6DeviceStatsMonitor.on_train_batch_³„Ä Íend/segment.all.peak‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339888498688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610807400830976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45359380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317135547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610807206988800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9499 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884240701721356e+09 _step‚779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137939058 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339718596608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 _runtime‚7569.9101853690’îö&ˆ$ðM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317155613 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610807400830976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179196489 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317155613 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137939060 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚537224375773184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45359380 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 179216553 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚551669177281536 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317135547 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610807206988800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9499 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602467512332288 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179216553 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 % _timestamp‚1.7884240701721356e+09 _step‚779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559618845066240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7949667784704 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137939060 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚602467488392192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚537224392812544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137939058 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137939058 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7919750331406 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559618170687488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8339718596608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80478132 _runtime‚7569.910185369 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚551669159278592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179196489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545144797581914 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚551669159278592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125837514 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545144126104590 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317135547 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7949011408896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125837056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚559618845066240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7920404769370 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8339888498688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613870343306240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45582608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8382138683904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180124125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310334464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 # _timestamp‚1.78842410615642e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310334464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613870537161728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126460974 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126460515 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673606 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605ÇŠ¥¼ê'—$äO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180124125 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚539914678170624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45582151 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605488398477824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562419635969024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚613870343306240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚539914695209984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45582608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8382138683904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚547874447406474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554430458182144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7989177786880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180124125 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562420310334464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80878366 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138625842 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138625844 # _timestamp‚1.78842410615642e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562420310334464 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚613870537161728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554430458182144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126460974 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126460515 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7960423673606 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605488374537728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ŽÆ]Õ¦2˜$- _wandbruntime‚7606  _runtime‚7606eµœñ&™$ëM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547930839449322 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382996610560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126473242 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613932864566784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670814 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613933045826048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554486811220992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚783 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45587170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884241068458617e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 _runtime‚7606.584011707 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138639860 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180142605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382839291392 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138639860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80886532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126473704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376‹q|cá#š$ðM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚562476808435712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990653590528 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138639860 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80886532 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318762399 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126473704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚547930839449322 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚547930167971998 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 trainer/global_step‚9549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8382996610560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126473242 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138639858 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚613932864566784 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7960585670814 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80886534 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚613933045826048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚539969582301184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138639858 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 318762399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7961240108778 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚554486829223936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7989997214720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554486829223936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554486811220992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605550025275392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚783 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180122541 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318782465 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554486811220992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚562477482814464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45587170 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884241068458617e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚462 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605550049215488 _runtime‚7606.584011707 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138639860 SEDeviceStatsMonitor.on_train_batch_end/allo‰¦Oæcation.small_pool.allocated‚ 180142605 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318782465 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8382839291392²*Lì2›$- _wandbruntime‚7606  _runtime‚7606;EYœ$Ó /loss/FRt3/assignment_loss‚2.770488935001826 2loss/SRqqt2/assignment_loss‚1.7404279747316915 2loss/SRqqt3/assignment_loss‚1.4209793546612848 ,loss/FBt2/detection_loss‚2.6607603260449 1loss/SRqqt1/detection_loss‚1.9237846441757984 1loss/SRqqt2/detection_loss‚2.4720754286935254 1loss/SRqqt3/detection_loss‚1.6903846744251878 epoch‚7 trainer/global_step‚9549 0loss/FRt4/assignment_loss‚0.9862746810540557 /loss/FRt4/detection_loss‚1.3407306100707501 /loss/FBt4/detection_loss‚0.7754706880637692 0loss/FRt1/assignment_loss‚2.8479243460155668 /loss/FRt2/assignment_loss‚2.785463459789753 0loss/FBt2/assignment_loss‚0.3421270142508937 1loss/FBt3/assignment_loss‚0.08476336202459477 .loss/FRt2/detection_loss‚2.911480564624071 /loss/FBt3/detection_loss‚2.1461244991847446 %loss/total_loss‚39.25803056113057 .loss/FBt1/detection_loss‚2.508358812994427 _runtime‚7606.585383896 /loss/FRt1/detection_loss‚2.4137267576796666 2loss/SRqqt4/assignment_loss‚0.7992905933908853 1loss/FBt4/assignment_loss‚0.10476498261442559 _step‚784 2loss/SRqqt1/assignment_loss‚1.2858487036174688 0loss/FBt1/assignment_loss‚0.1815248353717228 /loss/FRt3/detection_loss‚2.2451333239422007 1loss/SRqqt4/detection_loss‚0.8201219887073551 $ _timestamp‚1.788424106847652e+09H*¿ƒ^$Ø /loss/FRt1/detection_loss‚2.4137267576796666 2loss/SRqqt4/assignment_loss‚0.7992905933908853 1loss/FBt4/assignment_loss‚0.10476498261442559 _step‚784 2loss/SRqqt1/assignment_loss‚1.2858487036174688 0loss/FBt1/assignment_loss‚0.1815248353717228 /loss/FRt3/detection_loss‚2.2451333239422007 1loss/SRqqt4/detection_loss‚0.8201219887073551 $ _timestamp‚1.788424106847652e+09 /loss/FRt3/assignment_loss‚2.770488935001826 2loss/SRqqt2/assignment_loss‚1.7404279747316915 2loss/SRqqt3/assignment_loss‚1.4209793546612848 ,loss/FBt2/detection_loss‚2.6607603260449 1loss/SRqqt1/detection_loss‚1.9237846441757984 1loss/SRqqt2/detection_loss‚2.4720754286935254 1loss/SRqqt3/detection_loss‚1.6903846744251878 epoch‚7 trainer/global_step‚9549 0loss/FRt4/assignment_loss‚0.9862746810540557 /loss/FRt4/detection_loss‚1.3407306100707501 /loss/FBt4/detection_loss‚0.7754706880637692 0loss/FRt1/assignment_loss‚2.8479243460155668 /loss/FRt2/assignment_loss‚2.785463459789753 0loss/FBt2/assignment_loss‚0.3421270142508937 1loss/FBt3/assignment_loss‚0.08476336202459477 .loss/FRt2/detection_loss‚2.911480564624071 /loss/FBt3/detection_loss‚2.1461244991847446 %loss/total_loss‚39.25803056113057 .loss/FBt1/detection_loss‚2.508358812994427 _runtime‚7606.585383896¬3«K’ž$‚(:‡ ¸ßäÔ€çÆ~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.392' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0784 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IŠ¿MxŸ$‚(:í ¸ßäÔÕÍì€ disk.drbd1716.in‚0 disk.dm-0.in‚4.6875& disk.nvme0n1.out‚32054.47900390625 disk.rbd0.out‚0$ memory_percent‚93.15006732940674 network.recv‚ 2.800213e+06 disk.sda2.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚32054.47900390625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0* disk./.usagePercent‚17.461698661600803$ disk./.usageGB‚260.1919288635254( disk.nvme0c0n1.out‚32054.47900390625 disk.drbd2265.out‚0 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1122.29296875 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.sda3.out‚ 418.07421875 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2772.33984375 cpu‚25.328073606176424 network.sent‚ 2.2816414e+07 disk.sda2.in‚0 disk.dm-1.out‚ 418.07421875* proc.memory.percent‚16.921019554138184 disk.sda.out‚ 418.07421875†Âl’ $‚(:‡ ÇßäÔéï™~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u4FGt¡$‚(:é ÇßäÔÉ€ª€ disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚32114.25146484375* proc.memory.percent‚16.921019554138184( disk.nvme0c0n1.out‚32114.25146484375 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0* disk./.usagePercent‚17.462798213960365 disk.sda.out‚ 418.83203125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 418.83203125 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375& proc.memory.availableMB‚ 1122.53125 cpu‚25.3275000759024 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.6875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0# disk.dm-0.out‚32114.25146484375 disk.drbd1716.in‚0 network.sent‚ 2.2924229e+07 network.recv‚ 2.805571e+06% disk./.usageGB‚260.20831298828125 disk.sda3.out‚ 418.83203125 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875$ memory_percent‚93.14861297607422 disk.sda3.in‚ 126.77734375 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0# proc.memory.rssMB‚ 2772.33984375†äW2¢$- _wandbruntime‚7642  _runtime‚7642>XìÆ£$À $ _timestamp‚1.788424143004544e+09 _runtime‚7642.742079843 _step‚785 & lr-AdamW/pg1‚2.088952354987142e-05 & lr-AdamW/pg2‚2.088952354987142e-05 trainer/global_step‚9599 úIèˤ$Å & lr-AdamW/pg1‚2.088952354987142e-05 & lr-AdamW/pg2‚2.088952354987142e-05 trainer/global_step‚9599 $ _timestamp‚1.788424143004544e+09 _runtime‚7642.742079843 _step‚785‘}%=•2¥$- _wandbruntime‚7642  _runtime‚7642A°:Gç'¦$áO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181030139 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81286766 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45810010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8030819955200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127096774 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139326642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45810459 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616996023015936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550660489273882 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608570935361024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127097225 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8425111595008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273717248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550661160750998 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884241430053785e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163592704 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139326642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542659901737984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013014 _step‚786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557248110124544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45810010 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8030819955200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127096774 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139326642 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139326644 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45810459 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚616996023015936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚550660489273882 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542659884698624 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139326644 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557248128127488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557248110124544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚608570935361024 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8000604575258 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127097225 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81286764 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8425111595008 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 320356781 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚565278273717248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚565278273717248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚550661160750998 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 % _timestamp‚1.7884241430053785e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8030163592704 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139326642 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚542659901737984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚608570911420928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8001259013014 _step‚786 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557248110124544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557304463163392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8001421010222 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608632562158592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788424143698146e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127109835 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚565336120562688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550716209839406 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557304463163392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127109399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8425988607488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚7643.436245485 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚565336120562688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 trainer/global_step‚9599 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617058550766080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45814901 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617058727831040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960js¹yõ&ª$ïM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181068657 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚617058550766080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45814901 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚617058727831040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557304481166336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139340658 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181068657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8030983020544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031639396352 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45814467 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320389251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚608632586098688 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚565335446183936 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8002075448186 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚434 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚557304463163392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8001421010222 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320409317 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚608632562158592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788424143698146e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 320389251 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127109835 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139340660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚565336120562688 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139340660 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚550716209839406 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542714805868544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031639396352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557304463163392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320409317 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181048593 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181048593 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127109399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8425988607488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 _runtime‚7643.436245485 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚565336120562688 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81294934 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 trainer/global_step‚9599“[­ˆ³2«$- _wandbruntime‚7643  _runtime‚7643¦*óa¬$Û /loss/FRt1/detection_loss‚2.3813253343105316 /loss/FBt2/detection_loss‚3.5177467862764993 0loss/FRt2/assignment_loss‚3.8372473887034824 2loss/SRqqt2/assignment_loss‚1.8458293514216648 1loss/FBt4/assignment_loss‚0.16312502456108782 1loss/SRqqt2/detection_loss‚2.4885499792940475 /loss/FBt4/detection_loss‚0.7060763927154421 _step‚788 /loss/FRt1/assignment_loss‚2.927117520570755 1loss/FBt1/assignment_loss‚0.16673736739903405 /loss/FRt3/detection_loss‚2.2117384057153355 1loss/SRqqt1/detection_loss‚1.8582682606387646 1loss/SRqqt3/detection_loss‚1.8960765819657932 1loss/SRqqt4/detection_loss‚1.0943722058283656 &loss/total_loss‚43.449251014041856 0loss/FRt4/assignment_loss‚1.5255720459896585 /loss/FBt1/detection_loss‚2.2744031250476837 /loss/FBt3/detection_loss‚2.2138007964406694 trainer/global_step‚9599 _runtime‚7643.437285136 % _timestamp‚1.7884241436998558e+09 .loss/FRt2/detection_loss‚3.081810942717961 2loss/SRqqt3/assignment_loss‚1.4167153009636835 1loss/FBt3/assignment_loss‚0.12484016152796176 2loss/SRqqt4/assignment_loss‚1.0998297700527728 1loss/FBt2/assignment_loss‚0.42474870246708935 epoch‚7 /loss/FRt3/assignment_loss‚2.492279783162204 2loss/SRqqt1/assignment_loss‚1.8599045917867345 /loss/FRt4/detection_loss‚1.8411351944846304~Ÿ¬f­$à /loss/FBt3/detection_loss‚2.2138007964406694 trainer/global_step‚9599 _runtime‚7643.437285136 % _timestamp‚1.7884241436998558e+09 .loss/FRt2/detection_loss‚3.081810942717961 2loss/SRqqt3/assignment_loss‚1.4167153009636835 1loss/FBt3/assignment_loss‚0.12484016152796176 2loss/SRqqt4/assignment_loss‚1.0998297700527728 1loss/FBt2/assignment_loss‚0.42474870246708935 epoch‚7 /loss/FRt3/assignment_loss‚2.492279783162204 2loss/SRqqt1/assignment_loss‚1.8599045917867345 /loss/FRt4/detection_loss‚1.8411351944846304 /loss/FRt1/detection_loss‚2.3813253343105316 /loss/FBt2/detection_loss‚3.5177467862764993 0loss/FRt2/assignment_loss‚3.8372473887034824 2loss/SRqqt2/assignment_loss‚1.8458293514216648 1loss/FBt4/assignment_loss‚0.16312502456108782 1loss/SRqqt2/detection_loss‚2.4885499792940475 /loss/FBt4/detection_loss‚0.7060763927154421 _step‚788 /loss/FRt1/assignment_loss‚2.927117520570755 1loss/FBt1/assignment_loss‚0.16673736739903405 /loss/FRt3/detection_loss‚2.2117384057153355 1loss/SRqqt1/detection_loss‚1.8582682606387646 1loss/SRqqt3/detection_loss‚1.8960765819657932 1loss/SRqqt4/detection_loss‚1.0943722058283656 &loss/total_loss‚43.449251014041856 0loss/FRt4/assignment_loss‚1.5255720459896585 /loss/FBt1/detection_loss‚2.2744031250476837”ü÷õ’®$‚(:‡ ÖßäÔ¬’«~ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.142' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E!ý‚x¯$‚(:í ÖßäÔÇäÄ€ disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 419.47265625 disk.dm-1.in‚ 126.77734375 cpu‚25.326956369436804 proc.cpu.threads‚17* disk./.usagePercent‚17.462626688912426 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875# proc.memory.rssMB‚ 2772.33984375* proc.memory.percent‚16.921019554138184 disk.drbd2266.out‚0# disk.dm-0.out‚32164.11083984375 disk.dm-1.out‚ 419.47265625 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.sent‚ 2.2930296e+07$ disk./.usageGB‚260.2057571411133 disk.rbd1.out‚ 3.986328125 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0 disk.sda.out‚ 419.47265625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚32164.11083984375 disk.sda2.in‚0 disk.drbd1716.out‚0 network.recv‚ 2.809883e+06 disk.sda1.out‚0& disk.nvme0n1.out‚32164.11083984375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375$ memory_percent‚93.14877986907959) proc.memory.availableMB‚ 1122.50390625 disk.drbd1716.in‚0$+ê’°$‚(:‡ åßäÔÄ¿~ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…N‚u±$‚(:ê åßäÔû¢µ€ disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 network.recv‚ 2.816185e+06% disk.nvme0n1.out‚32244.0927734375 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875$ memory_percent‚93.14730167388916' disk.nvme0c0n1.out‚32244.0927734375 disk.sda.in‚ 126.77734375) proc.memory.availableMB‚ 1122.74609375# proc.memory.rssMB‚ 2772.33984375* proc.memory.percent‚16.921019554138184 disk.drbd1716.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚32244.0927734375 disk.sda.out‚ 420.65234375 disk.sda1.in‚0 network.sent‚ 2.3038838e+07 disk.dm-1.out‚ 420.65234375 cpu‚25.326417054324434 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0$ disk./.usageGB‚260.2172622680664 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0 disk.sda3.out‚ 420.65234375 disk.drbd2265.out‚0* disk./.usagePercent‚17.463398807635695 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17+«{â2²$- _wandbruntime‚7679  _runtime‚7679Ð#¥É³$à ' lr-AdamW/pg1‚2.0094026770973047e-05 ' lr-AdamW/pg2‚2.0094026770973047e-05 trainer/global_step‚9649 % _timestamp‚1.7884241794681728e+09 _runtime‚7679.205691405 _step‚7895SVÃδ$È ' lr-AdamW/pg1‚2.0094026770973047e-05 ' lr-AdamW/pg2‚2.0094026770973047e-05 trainer/global_step‚9649 % _timestamp‚1.7884241794681728e+09 _runtime‚7679.205691405 _step‚789•Ðjø2µ$- _wandbruntime‚7679  _runtime‚7679Î H2ç'¶$áO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚7679.206558221 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127732067 % _timestamp‚1.7884241794688632e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620121889998848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620121702434816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46037349 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8468417754624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚790 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8468254130688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140027444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911465472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0‹ÝâN·$æO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚560065780069888 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 322003673 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321983633 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚560065762066944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8071805761024 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚611653472244224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81695164 MGDeviceStatsMonitbÉuО#or.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181956191 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553446531141290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _runtime‚7679.206558221 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8071805761024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140027442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚568137585830912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127732067 % _timestamp‚1.7884241794688632e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚620121889998848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚620121702434816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚560065762066944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140027442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46037349 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8041439914666 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8071149398528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚560065780069888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8468417754624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545405108265984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 _step‚790 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8468254130688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140027444 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚568136911465472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚611653448304128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8042094352422 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚568136911465472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81695166 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚568137585830912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181976229 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0–Rž]†2¸$- _wandbruntime‚7679  _runtime‚7679/k2Xñ&¹$ëM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071968826368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8042910787594 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46041323 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 _runtime‚7679.897128161 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127745105 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81703334 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974645 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884241801589863e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568194758310912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 trainer/global_step‚9649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994709 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127744655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620184200730624 _step‚791 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968826368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140041460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46041771 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036169 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184138š~“ö&º$ðM SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140041460 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46041771 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 322036169 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚553502923184138 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8071968826368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140041458 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8042910787594 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚560122133108736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46041323 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8042256349630 _runtime‚7679.897128161 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚611715122981888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚560122115105792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127745105 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 322016103 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81703332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚568194758310912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚568194083932160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81703334 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974645 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 % _timestamp‚1.7884241801589863e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚568194758310912 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140041458 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚560122115105792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 322036169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 trainer/global_step‚9649 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181994709 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127744655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8072625202176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚620184200730624 _step‚791 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8071968826368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952—&éÄÌ2»$- _wandbruntime‚7679  _runtime‚7679 eéN]¼$× /loss/FRt1/detection_loss‚2.0190209187567234 _step‚792 2loss/SRqqt2/assignment_loss‚1.7133328696769294 1loss/FBt1/assignment_loss‚0.34074064819142225 0loss/FBt2/assignment_loss‚0.2881159782506681 -loss/FRt3/detection_loss‚2.38230536759577 /loss/FRt4/detection_loss‚1.6357064880079255 1loss/SRqqt3/detection_loss‚1.8146706780172743 /loss/FBt4/detection_loss‚0.4996443282835549 _runtime‚7679.898134781 0loss/FRt3/assignment_loss‚2.6864473860372198 /loss/FBt1/detection_loss‚1.9752139806747437 .loss/FRt2/assignment_loss‚3.34092431232847 2loss/SRqqt1/assignment_loss‚1.3227162707286577 1loss/SRqqt1/detection_loss‚1.8283304255455732 0loss/SRqqt2/detection_loss‚2.456016920732729 .loss/FBt3/detection_loss‚5.397363662719727 2loss/SRqqt4/assignment_loss‚0.9479326526943623 3loss/FBt3/assignment_loss‚0.0060011413879692554 1loss/FBt4/assignment_loss‚0.07390507032324388 .loss/FBt2/detection_loss‚2.129464882115523 trainer/global_step‚9649 % _timestamp‚1.7884241801606662e+09 epoch‚7 1loss/SRqqt4/detection_loss‚0.9169588321237825 0loss/FRt4/assignment_loss‚1.5223613235185731 %loss/total_loss‚42.13556951805597 0loss/FRt1/assignment_loss‚2.6663325863579908 2loss/SRqqt3/assignment_loss‚1.0763350337708615 /loss/FRt2/detection_loss‚3.0957277602162856倫b½$Ü 2loss/SRqqt3/assignment_loss‚1.0763350337708615 /loss/FRt2/detection_loss‚3.0957277602162856 /loss/FRt1/detection_loss‚2.0190209187567234 _step‚792 2loss/SRqqt2/assignment_loss‚1.7133328696769294 1loss/FBt1/assignment_loss‚0.34074064819142225 0loss/FBt2/assignment_loss‚0.2881159782506681 -loss/FRt3/detection_loss‚2.38230536759577 /loss/FRt4/detection_loss‚1.6357064880079255 1loss/SRqqt3/detection_loss‚1.8146706780172743 /loss/FBt4/detection_loss‚0.4996443282835549 _runtime‚7679.898134781 0loss/FRt3/assignment_loss‚2.6864473860372198 /loss/FBt1/detection_loss‚1.9752139806747437 .loss/FRt2/assignment_loss‚3.34092431232847 2loss/SRqqt1/assignment_loss‚1.3227162707286577 1loss/SRqqt1/detection_loss‚1.8283304255455732 0loss/SRqqt2/detection_loss‚2.456016920732729 .loss/FBt3/detection_loss‚5.397363662719727 2loss/SRqqt4/assignment_loss‚0.9479326526943623 3loss/FBt3/assignment_loss‚0.0060011413879692554 1loss/FBt4/assignment_loss‚0.07390507032324388 .loss/FBt2/detection_loss‚2.129464882115523 trainer/global_step‚9649 % _timestamp‚1.7884241801606662e+09 epoch‚7 1loss/SRqqt4/detection_loss‚0.9169588321237825 0loss/FRt4/assignment_loss‚1.5223613235185731 %loss/total_loss‚42.13556951805597 0loss/FRt1/assignment_loss‚2.6663325863579908˜ñA µ’¾$‚(:‡ ôßäÔöø‘ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚125.301' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D½‡¿$‚(:â ôßäÔ¨ÄÇ€ disk.sda2.out‚0# disk.dm-0.out‚32296.08935546875 disk.dm-1.out‚ 421.34375 disk.sda.out‚ 421.34375& disk.nvme0n1.out‚32296.08935546875& proc.memory.availableMB‚ 1122.28125 network.sent‚ 2.3045139e+07 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 cpu‚25.325878029431934* disk./.usagePercent‚17.462734212076807( disk.nvIF$áæme0c0n1.out‚32296.08935546875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚260.20735931396484 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875$ memory_percent‚93.15013885498047 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.sda3.out‚ 421.34375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2772.33984375 proc.cpu.threads‚17 network.recv‚ 2.821032e+06 disk.drbd2080.out‚0 disk.rbd1.out‚ 3.986328125 disk.sda1.out‚0* proc.memory.percent‚16.921019554138184Ë*kk’À$‚(:‡ ƒàäÔŠÁ…~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.4458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 h–…xÁ$‚(:í ƒàäÔÎ×ÿ disk.drbd1716.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2772.33984375 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0& disk.nvme0n1.out‚32364.83447265625 disk.nvme0c0n1.in‚4.6875% disk./.usageGB‚260.24203872680664 disk.sda.in‚ 126.77734375 proc.cpu.threads‚17 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 421.96484375 disk.drbd2266.in‚0 disk.sda.out‚ 421.96484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 421.96484375$ memory_percent‚93.14920902252197# disk.dm-0.out‚32364.83447265625 disk.drbd2266.out‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.921019554138184* disk./.usagePercent‚17.465061576570584 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0 disk.rbd0.in‚0 cpu‚25.325313543058503 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚32364.83447265625) proc.memory.availableMB‚ 1122.43359375 network.sent‚ 2.3153073e+07 network.recv‚ 2.826292e+06 disk.rbd1.in‚0AüJÜ’Â$‚(:‡ ’àäÔ鬔~ gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œA5©cÃ$‚(:Ø ’àäÔ’ˆ› disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 network.sent‚ 2.3159121e+07 disk.dm-1.out‚ 422.578125 cpu‚25.33433700337159 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 network.recv‚ 2.830643e+06 disk.drbd1716.out‚0# disk.nvme0n1.out‚32411.08203125 disk.sda2.in‚0% proc.memory.availableMB‚ 1122.1875# proc.memory.rssMB‚ 2772.33984375 disk.drbd1716.in‚0 disk.sda3.out‚ 422.578125 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 422.578125* proc.memory.percent‚16.921019554138184$ disk./.usageGB‚260.2197151184082 disk.drbd2265.out‚0 disk.dm-0.out‚32411.08203125 proc.cpu.threads‚17 disk.nvme0n1.in‚4.6875% disk.nvme0c0n1.out‚32411.08203125 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.986328125 disk.sda2.out‚0) disk./.usagePercent‚17.46356342048021 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375$ memory_percent‚93.15071105957031 disk.rbd0.in‚0 disk.rbd0.out‚0Énùù’Ä$‚(:‡ ¡àäÔÌ‘~ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚183.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.7612 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¨ãËvÅ$‚(:ë ¡àäԩŃ€ disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 422.97265625# disk.dm-0.out‚32484.32275390625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0$ memory_percent‚93.14916133880615 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 422.97265625* proc.memory.percent‚16.921019554138184 proc.cpu.threads‚17 network.sent‚ 2.3165015e+07) disk./.usagePercent‚17.46519598052606 disk.sda.out‚ 422.97265625 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚32484.32275390625 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1122.44140625 network.recv‚ 2.834707e+06 disk.sda1.out‚0& disk.nvme0n1.out‚32484.32275390625 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 3.986328125# proc.memory.rssMB‚ 2772.33984375$ disk./.usageGB‚260.2440414428711 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 cpu‚25.34289019330417™?B’Æ$‚(:‡ °àäÔõÔÕ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š ºœuÇ$‚(:ê °àäÔµ°æ* disk./.usagePercent‚17.465096137587707 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0$ disk./.usageGB‚260.2425537109375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0& disk.nvme0n1.out‚32534.54833984375 disk.dm-1.out‚ 423.67578125 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 network.recv‚ 2.839634e+06 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.921019554138184 cpu‚25.35267796182792 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2772.33984375 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 423.67578125 disk.dm-0.in‚4.6875# disk.dm-0.out‚32534.54833984375 network.sent‚ 2.317134e+07 disk.drbd2266.out‚0 disk.sda.out‚ 423.67578125 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚32534.54833984375$ memory_percent‚93.14913749694824( proc.memory.availableMB‚ 1122.4453125 disk.sda2.in‚0 proc.cpu.threads‚17T™ 2È$- _wandbruntime‚7743  _runtime‚7743œsÚ­M*É$ÇT _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚9671 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚563166571374592 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 323637913 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 182758607 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚614927326212096 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚548397280444416 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚ 7743.11858158 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137186816 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291686 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚157 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128560965 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884242433806295e+09 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245438 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8506150572544 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622704128 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚556476006245438 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚414 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 8108562162688 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8506150572544 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚563166553371648 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182742630 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚571274622704128 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 140879306 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _runtime‚ 7743.11858158 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 137186816 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8108069332480 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8108562162688 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚556475498291686 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚571275133537280 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 140879306 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚157 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚614927350152192 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 128560965 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚430 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 323637913 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 182742630 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 % _timestamp‚1.7884242433806295e+09 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182749932 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚C‚?°2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8506515699200 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚571312328692736 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚563203401403392 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚563203383400448 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137186816 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 128572039 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8506628945920 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 140890112 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128571882 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 323640042 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656021 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817859584 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8108434459136 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚571312328692736 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚563203401403392 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚563203383400448 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8108434459136 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 137186816 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚571312328692736 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 140890110 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 128572039 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8506628945920 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 140890112 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 128571882 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 323640042 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323656021 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 323640042 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚614965201945600 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚571311817859584 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚563203383400448 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8108434459136 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 8108927289344 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182749932 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 9148669952 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 140890112 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182765909 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8506515699200 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚556511663614334 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚571311817859584 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚30 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18832597136497498 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.583062469959259 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1020379289984703 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29234611988067627 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.507758617401123 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6025298237800598 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5662931799888611 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0765988901257515 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5946320295333862 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1597224473953247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5714734196662903 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5603848099708557 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920694828033447 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965061545372009 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10822924971580505 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1260095089673996 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 _runtime‚7756.649522793 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5506117939949036 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5699735879898071 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_2‚0.7816391587257385 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15899211168289185 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5426414608955383 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5827315449714661 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1912095546722412 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37310636043548584 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5601744651794434 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12950976192951202 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030626168474555016 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.46152299642562866 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5836533904075623 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023739119991660118 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,particle/sensitivity‚0.31048068404197693 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5925925970077515 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_3_of_5‚0.12860342860221863 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_6_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.300148606300354 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5010879635810852 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4888888895511627 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚9671 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13418170809745789 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12111777067184448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5738094449043274 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt*FBt/event_propoê?-Áÿ\rtion‚0.2310267835855484 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10892856866121292 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5854983925819397 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190246820449829 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46884533762931824 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 _step‚795 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.540915846824646 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08932947367429733 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4673110842704773 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881137013435364 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6191536784172058 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0708671286702156 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013460778631269932 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5570641160011292 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6010147333145142 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 particle/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.107142873108387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5490841269493103 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5478770732879639 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07229669392108917 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997493326663971 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5619543790817261 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012527617625892162 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1331593543291092 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.004926473367959261 ,jet/accuracy_4_of_5‚0.013978494331240654 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.977729856967926 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12428028136491776 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27037397027015686 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25749123096466064 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4167312979698181 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0795019268989563 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944715976715 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6124998927116394 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 +jet/accuracy_2_of_2‚0.11216423660516739 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008616780862212181 jet/accuracy_5_of_5‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38351020216941833 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919263124465942 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23716644942760468 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3933035433292389 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4698198139667511 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1983024775981903 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5593409538269043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5056019425392151 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9908019304275513 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_6‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5421268343925476 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877504110336304 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03424433246254921 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6105304956436157 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚42.77785049720536 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982366681098938 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1725338250398636 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3146592378616333 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5430271625518799 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5723135471343994 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5671678781509399 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11742424964904785 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1909750998020172 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10555555671453476 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 (particle/accuracy‚0.8490204215049744 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03897012397646904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.503801167011261 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5734694600105286 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6996182203292847 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878649115562439 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5367284417152405 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5290868282318115 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196218252182007 +jet/accuracy_2_of_5‚0.49432283639907837 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *jet/accuracy_1_of_6‚0.9615384340286255 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15899211168289185 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12139392644166946 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.277777761220932 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46895426511764526 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11530321091413498 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 +jet/accuracy_3_of_6‚0.19230769574642181 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24426713585853577 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15687397122383118 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900980591773987 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12671828269958496 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5873016119003296 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884242569097795e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3634154498577118 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20025618374347687 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25152403116226196 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03312554955482483 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34869006276130676 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1828038990497589 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33348220586776733 particle/accuracy_2_of_3‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2611502408981323 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4124051332473755 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.550983726978302 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4546046853065491 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.588723361492157 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5895832777023315 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01757388934493065 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2608529031276703 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5226190686225891 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5709876418113708 jet/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17575936019420624 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3225758671760559 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5747126936912537 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 *jet/accuracy_1_of_5‚0.9565540552139282 jet/accuracy_5_of_6‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9264434576034546 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.281598836183548 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_1‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.544289767742157 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09344129264354706 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9736143350601196 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09321697801351547 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12032591551542282 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22758620977401733 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5137308835983276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9816285967826843 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4205426573753357 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 jet/accuracy_3_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895991683006287 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10094106197357178 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02358732372522354 particle/accuracy_4_of_4‚1 'particle/f_score‚0.3909350037574768 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776114225387573 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5725405812263489 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5723135471343994 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07957801222801208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1748243123292923 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04853867366909981 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3583332896232605 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9857363104820251 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15870152413845062 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6384081840515137 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.04423868656158447=ÜÈñó"Ò$‡§ ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.004926473367959261 ,jet/accuracy_4_of_5‚0.013978494331240654 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.977729856967926 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12428028136491776 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27037397027015686 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.25749123096466064 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4167312979698181 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0795019268989563 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3256944715976715 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6124998927116394 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 +jet/accuracy_2_of_2‚0.11216423660516739 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008616780862212181 jet/accuracy_5_of_5‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38351020216941833 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9919263124465942 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333730697632 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23716644942760468 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3933035433292389 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.4698198139667511 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1983024775981903 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5593409538269043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5056019425392151 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9908019304275513 particle/accuracy_3_of_3‚1 particle/accuracy_4_of_6‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 jet/accuracy_2_of_7‚0.5 jet/accuracy_7_of_7‚0 particle/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5421268343925476 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9744047522544861 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877504110336304 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03424433246254921 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6105304956436157 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 )loss/val_total_loss‚42.77785049720536 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.982366681098938 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1725338250398636 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3146592378616333 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5430271625518799 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5723135471343994 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5671678781509399 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.11742424964904785 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1909750998020172 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10555555671453476 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 (particle/accuracy‚0.8490204215049744 particle/accuracy_3_of_6‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03897012397646904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.503801167011261 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5734694600105286 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6996182203292847 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878649115562439 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5367284417152405 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5290868282318115 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196218252182007 +jet/accuracy_2_of_5‚0.49432283639907837 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 *jet/accuracy_1_of_6‚0.9615384340286255 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15899211168289185 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12139392644166946 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.277777761220932 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.46895426511764526 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11530321091413498 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1111111119389534 +jet/accuracy_3_of_6‚0.19230769574642181 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24426713585853577 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15687397122383118 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9900980591773987 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_6øá,‚›p_of_7‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9760233759880066 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12671828269958496 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5873016119003296 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 % _timestamp‚1.7884242569097795e+09 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3634154498577118 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20025618374347687 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25152403116226196 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03312554955482483 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34869006276130676 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1828038990497589 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.33348220586776733 particle/accuracy_2_of_3‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2611502408981323 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4124051332473755 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.550983726978302 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.4546046853065491 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.588723361492157 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5895832777023315 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01757388934493065 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2608529031276703 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5226190686225891 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5709876418113708 jet/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17575936019420624 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3225758671760559 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5747126936912537 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 *jet/accuracy_1_of_5‚0.9565540552139282 jet/accuracy_5_of_6‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9264434576034546 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.281598836183548 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_1‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.544289767742157 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09344129264354706 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9736143350601196 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09321697801351547 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12032591551542282 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22758620977401733 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5137308835983276 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9816285967826843 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4205426573753357 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 jet/accuracy_3_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895991683006287 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10094106197357178 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.02358732372522354 particle/accuracy_4_of_4‚1 'particle/f_score‚0.3909350037574768 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_6_of_6‚0 jet/accuracy_5_of_7‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776114225387573 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5725405812263489 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5723135471343994 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07957801222801208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1748243123292923 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04853867366909981 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3583332896232605 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9857363104820251 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15870152413845062 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6384081840515137 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.04423868656158447 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08865249156951904 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20200039446353912 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5192946791648865 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20566794276237488 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35718485713005066 *particle/specificity‚0.948688268661499 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209877610206604 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09010190516710281 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5674975514411926 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41346070170402527 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5263298153877258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5076969861984253 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_3‚0.24232877790927887 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2675926089286804 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.049158453941345215 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9853219389915466 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_7‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5041187405586243 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1534099578857422 *jet/accuracy_1_of_3‚0.8882095813751221 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1088889017701149 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12008833140134811 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12729886174201965 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45920073986053467 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.14414414763450623 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5657670497894287 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5249631404876709 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_4‚0.38940513134002686 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_6‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953427314758301 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5496376752853394 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002824336290359497 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9844093322753906 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08396635949611664 *jet/accuracy_2_of_6‚0.6089743971824646 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.013071895577013493 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6187728643417358 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3402557671070099 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.49261075258255005 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333730697632 epoch‚7 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18832597136497498 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.583062469959259 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1020379289984703 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29234611988067627 8"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.507758617401123 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6025298237800598 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5662931799888611 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0765988901257515 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5946320295333862 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1597224473953247 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5714734196662903 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5603848099708557 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920694828033447 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9965061545372009 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10822924971580505 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1260095089673996 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 _runtime‚7756.649522793 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5506117939949036 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0172955971211195 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5699735879898071 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 *jet/accuracy_1_of_2‚0.7816391587257385 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15899211168289185 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5426414608955383 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5827315449714661 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1912095546722412 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37310636043548584 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5601744651794434 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12950976192951202 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030626168474555016 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.013071895577013493 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.46152299642562866 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 particle/accuracy_4_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6578947305679321 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5836533904075623 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023739119991660118 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,particle/sensitivity‚0.31048068404197693 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5925925970077515 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 +jet/accuracy_3_of_5‚0.12860342860221863 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_6_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.300148606300354 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5010879635810852 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4888888895511627 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 trainer/global_step‚9671 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13418170809745789 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12111777067184448 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5738094449043274 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10892856866121292 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5854983925819397 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.111111119389534 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190246820449829 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.46884533762931824 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 _step‚795 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.540915846824646 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08932947367429733 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4673110842704773 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_7‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9881137013435364 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6191536784172058 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0708671286702156 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3598484992980957 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013460778631269932 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.05263157933950424 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5570641160011292 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6010147333145142 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 particle/accuracy_1_of_6‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0071428571827709675 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.107142873108387 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5490841269493103 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5478770732879639 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07229669392108917 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997493326663971 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5619543790817261 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012527617625892162 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.625 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1331593543291092› 9’Ó$‚(:‡ ÎàäÔ¦¹Ž~ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UmñélÔ$‚(:á ÎàäÔ©®Ÿ€ disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.rbd1.out‚ 3.986328125$ memory_percent‚91.29228591918945% disk./.usageGB‚260.29285430908203% disk.nvme0c0n1.out‚33590.51171875 disk.sda.in‚ 126.77734375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 424.87890625" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 network.sent‚ 2.3284986e+07 network.recv‚ 2.849441e+06 disk.sda3.out‚ 424.87890625 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1426.671875 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.dm-0.in‚4.6875 disk.dm-0.out‚33590.51171875 disk.sda.out‚ 424.87890625 disk.sda1.out‚0 disk.drbd1716.out‚0 cpu‚25.36082320950462* disk./.usagePercent‚17.468471852934197# disk.nvme0n1.out‚33590.51171875 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0gtA2Õ$- _wandbruntime‚7780  _runtime‚7780¨ØñÃÆÖ$À _runtime‚ 7780.82044982 _step‚796 & lr-AdamW/pg1‚1.931191944502304e-05 & lr-AdamW/pg2‚1.931191944502304e-05 trainer/global_step‚9699 % _timestamp‚1.7884242810829215e+09ªB³Ë×$Å _runtime‚ 7780.82044982 _step‚796 & lr-AdamW/pg1‚1.931191944502304e-05 & lr-AdamW/pg2‚1.931191944502304e-05 trainer/global_step‚9699 % _timestamp‚1.7884242810829215e+09œ˜“8?2Ø$- _wandbruntime‚7780  _runtime‚7780ƘÜÒ:Ù$áO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8135811536896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46542400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 HBDeviceStatsMonito6º•Ë­ r.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573338860998144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155174400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183364244 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129056144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565203705823744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141409020 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884242810836236e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46542826 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚550373821641728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625656587792384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7780.821367712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617122180338176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617122156398080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625656766967808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚565203705823744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756@‰ºPì'Ú$æO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46542400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573338860998144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8135155174400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573338860998144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 183364244 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129056144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324753226 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8135155174400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚565203705823744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324773266 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141409020 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 183344206 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚550373804602368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 % _timestamp‚1.7884242810836236e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135811536896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46542826 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183344206 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚573339535363584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚426 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚550373821641728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚625656587792384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚7780.821367712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚617122180338176 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617122156398080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚625656766967808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚565203705823744 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141409022 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8135811536896 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚558479675833640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚9699<ïÁg2Û$- _wandbruntime‚7781  _runtime‚7781§>|.ñ&Ü$ëM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚565260076865536 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚625719088404480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚798 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8106016189120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8135974602240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _runtime‚7781.509598423 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8535281268736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82521484 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129068335 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630978048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183382724 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362660 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚573396707843584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573396033464832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141423036 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707843584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617183807135744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46547294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625719280149504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324805762 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183382724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141423036 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚573396707843584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚617183807135744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚558535396399372 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46547294 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚565260058862592 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8106670627084 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚625719280149504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚558534724922048 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324805762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141423038 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚617183831075840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peaÖÜ"ÏÕk‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129068335 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183362660 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8136630978048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183382724 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183362660 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324785696 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135974602240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324785696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141423036 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414ž vj¸2Þ$- _wandbruntime‚7781  _runtime‚7781…üòÒ[ß$Õ 0loss/FBt4/assignment_loss‚0.1774735597959936 epoch‚8 $ _timestamp‚1.788424281773266e+09 _runtime‚7781.510987093 _step‚799 0loss/FRt2/assignment_loss‚3.0015361339617996 0loss/FRt4/assignment_loss‚1.3965222993675543 0loss/FBt2/assignment_loss‚0.2538297207350695 /loss/FRt2/detection_loss‚2.6185124898568177 /loss/FBt2/detection_loss‚2.7707920521497726 /loss/FBt3/detection_loss‚1.9842432865074702 .loss/FRt3/detection_loss‚2.443778497832162 1loss/SRqqt2/detection_loss‚2.2550831777708873 0loss/FRt3/assignment_loss‚2.7933365872928073 2loss/SRqqt4/assignment_loss‚0.9508629402139185 /loss/FRt1/detection_loss‚2.2229936639467875 1loss/SRqqt3/assignment_loss‚1.857899777053131 1loss/SRqqt1/detection_loss‚1.8608774387836455 1loss/SRqqt4/detection_loss‚0.9406196203764895 /loss/FBt1/detection_loss‚2.2541595697402954 %loss/total_loss‚41.21001344340468 trainer/global_step‚9699 2loss/SRqqt2/assignment_loss‚1.8096829504839012 /loss/FRt4/detection_loss‚1.5925121131752218 1loss/SRqqt3/detection_loss‚2.0239822642670737 .loss/FBt4/detection_loss‚0.405746179297985 0loss/FRt1/assignment_loss‚3.0565477384461297 2loss/SRqqt1/assignment_loss‚1.6290663712937385 0loss/FBt1/assignment_loss‚0.6288373111747205 0loss/FBt3/assignment_loss‚0.2811176998813088@­’Ñ`à$Ú 0loss/FRt3/assignment_loss‚2.7933365872928073 2loss/SRqqt4/assignment_loss‚0.9508629402139185 /loss/FRt1/detection_loss‚2.2229936639467875 1loss/SRqqt3/assignment_loss‚1.857899777053131 1loss/SRqqt1/detection_loss‚1.8608774387836455 1loss/SRqqt4/detection_loss‚0.9406196203764895 /loss/FBt1/detection_loss‚2.2541595697402954 %loss/total_loss‚41.21001344340468 trainer/global_step‚9699 2loss/SRqqt2/assignment_loss‚1.8096829504839012 /loss/FRt4/detection_loss‚1.5925121131752218 1loss/SRqqt3/detection_loss‚2.0239822642670737 .loss/FBt4/detection_loss‚0.405746179297985 0loss/FRt1/assignment_loss‚3.0565477384461297 2loss/SRqqt1/assignment_loss‚1.6290663712937385 0loss/FBt1/assignment_loss‚0.6288373111747205 0loss/FBt3/assignment_loss‚0.2811176998813088 0loss/FBt4/assignment_loss‚0.1774735597959936 epoch‚8 $ _timestamp‚1.788424281773266e+09 _runtime‚7781.510987093 _step‚799 0loss/FRt2/assignment_loss‚3.0015361339617996 0loss/FRt4/assignment_loss‚1.3965222993675543 0loss/FBt2/assignment_loss‚0.2538297207350695 /loss/FRt2/detection_loss‚2.6185124898568177 /loss/FBt2/detection_loss‚2.7707920521497726 /loss/FBt3/detection_loss‚1.9842432865074702 .loss/FRt3/detection_loss‚2.443778497832162 1loss/SRqqt2/detection_loss‚2.2550831777708873Ÿæ¹k‘’á$‚(:‡ ÝàäÔ°ï¬~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.101' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À‘ ƒnâ$‚(:ã ÝàäÔíÉ·€ proc.cpu.threads‚17# disk.nvme0n1.out‚33649.88671875 disk.sda2.out‚0 disk.drbd2265.out‚0$ memory_percent‚91.29116535186768* proc.memory.percent‚16.921377182006836 disk.drbd2266.out‚0 disk.sda3.out‚ 425.19921875 disk.dm-0.out‚33649.88671875 network.recv‚ 2.855452e+06 disk.drbd1716.out‚0 network.sent‚ 2.340924e+07* disk./.usagePercent‚17.466499570890413 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1426.85546875 disk.sda.out‚ 425.19921875 disk.rbd0.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 425.19921875 cpu‚25.360225053420756 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0% disk./.usageGB‚260.26346588134766 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875% disk.nvme0c0n1.out‚33649.88671875 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2772.3984375bõòK’ã$‚(:‡ ìàäÔÁ—™~ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.804' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.9608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/}+èYä$‚(:Î ìàäÔ»³ disk.nvme0c0n1.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.sda.out‚425.75 disk.dm-0.in‚4.6875 disk.dm-1.out‚425.75 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2080.in‚0$ disk./.usageGB‚260.2762031555176% disk.nvme0c0n1.out‚33696.60546875 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0n1.out‚33696.60546875 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 3.986328125( proc.memory.availableMB‚ 1426.3671875 disk.dm-0.out‚33696.60546875 network.sent‚ 2.3518001e+07 network.recv‚ 2.861816e+06) disk./.usagePercent‚17.46735438004724 disk.sda3.out‚425.75 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.rbd0.in‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0$ memory_percent‚91.29414558410645 cpu‚25.35962604587267GQd’å$‚(:‡ ûàäÔ©«€~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚169.286' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.8572 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07þ1mæ$‚(:â ûàäÔëÊ‹€ network.sent‚ 2.3523099e+07 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚ 426.15625 disk.drbd2265.out‚0 disk.dm-1.out‚ 426.15625* disk./.usagePercent‚17.467562514172577% disk./.usageGB‚260.27930450439453 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚33751.06005859375 disk.drbd2266.in‚0 disk.drbd2266.out‚0' proc.memory.availableMB‚ 1426.859375 disk.nvme0n1.in‚4.6875 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0$ memory_percent‚91.29114151000977 disk.rbd1.in‚0( disk.nvme0c0n1.out‚33751.06005859375 disk.sda2.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚33751.06005859375 disk.sda.out‚ 426.15625 cpu‚25.359008599918734 network.recv‚ 2.864534e+06 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 3.986328125 disk.sda1.in‚0* proc.memory.percent‚16.921377182006836W52ç$- _wandbruntime‚7817  _runtime‚7817gíú`Éè$à _step‚800 ' lr-AdamW/pg1‚1.8543364562015936e-05 ' lr-AdamW/pg2‚1.8543364562015936e-05 trainer/global_step‚9749 % _timestamp‚1.7884243174081314e+09 _runtime‚7817.145640538fÛÎé$È % _timestamp‚1.7884243174081314e+09 _runtime‚7817.145640538 _step‚800 ' lr-AdamW/pg1‚1.8543364562015936e-05 ' lr-AdamW/pg2‚1.8543364562015936e-05 trainer/global_step‚9749 ÕôDK2ê$- _wandbruntime‚7817  _runtime‚7817K—4ræ'ë$àO $ _timestamp‚1.788424317408823e+09 _step‚801 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568021357766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129687686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚7817.146514617 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46766394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8577196482560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553119011130368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561265046223932 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568021357766144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142109822 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628781889763840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184270258 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620204717221376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173111808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184290296 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8176797342720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚576198173111808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8176797342720 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚568021357766144 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129687686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚7817.146514617 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184290296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46766394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8577196482560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚553119011130368 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82921718 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚561265046223932 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326400118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚568021357766144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚576197498746368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚568021375769088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142109822 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142109820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.·iõèall.current‚ 671477116 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚628781889763840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8146689531320 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 184270258 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620204693281280 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620204717221376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚576198173111808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0¡.5.Q2í$- _wandbruntime‚7817  _runtime‚7817~HVð&î$êM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788424318091105e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129700411 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7817.829279516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412548 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671213056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960408064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620266344018944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628844590694912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960408064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129700869 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628844401047040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561321438266780 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620266367959040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184308776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616783872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553173915260928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46770527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142123838›Ìáõ&ï$ïM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326412548 _step‚802 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46770983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184288712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129700411 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326432614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7817.829279516 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326412548 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚576254671213056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8176960408064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚568077710804992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620266344018944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚561320766789456 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326432614 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚628844590694912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8176960408064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184288712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 129700869 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628844401047040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561321438266780 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚576255345591808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚568077728807936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620266367959040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184308776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8177616783872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553173915260928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46770527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142123838 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184308776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 $ _timestamp‚1.788424318091105e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚576255345591808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142123836 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570839027711488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83330118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8620220168192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631907280417280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130321866 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46991301 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432972 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8217126786048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185216348 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564051759568456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185¢ÏÄü/216348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623287254104576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83330116 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631907474272768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚570839009708544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8217783148544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142810620 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579056810860032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130321417 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579056136494592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026970 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚7853.913192259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810860032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185196310 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46991748 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783148544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8620050252800 % _timestamp‚1.7884243541754363e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564051088091340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128h-vÚì'ü$æO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185216348 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚564051759568456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185216348 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚623287254104576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚555864217658368 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83330116 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚631907474272768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚570839009708544 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8217126786048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8217783148544 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142810620 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570839027711488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142810622 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚579056810860032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _step‚805 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142810622 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130321417 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚579056136494592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 328026970 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 _runtime‚7853.913192259 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚579056810860032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185196310 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚555864234697728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142810620 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46991748 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8217783148544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8620050252800 % _timestamp‚1.7884243541754363e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚564051088091340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328006930 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570839027711488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83330118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8620220168192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚631907280417280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130321866 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46991301 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8186870432972 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚579056136494592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570839009708544 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8217126786048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8187524870728 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¥œ4¤‰2ý$- _wandbruntime‚7854  _runtime‚7854C S•ñ&þ$ëM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 trainer/global_step‚9799 _step‚806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969779184128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7854.606958454 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328039400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8621070281216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234828 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46995698 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130333982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚012’‡Ÿÿ$ðM trainer/global_step‚9799 _step‚806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8218602589696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8218602589696 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚631969779184128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚564106808656864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570895380750336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚441 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7854.606958454 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328039400 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142824636 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚579113983340032 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚570895380750336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83338284 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142824638 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8621070281216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚570895362747392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚555919138828288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185234828 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46995698 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚579113308961280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚564107480134188 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328059466 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142824638 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142824636 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130333982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KåS¯ˆW EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8187686867936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185214764 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570895362747392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8217946213888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚579113983340032 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 328039400 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185234828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚555919121788928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430¦35„·2€%- _wandbruntime‚7854  _runtime‚7854³åª¶^%Ø 1loss/SRqqt2/detection_loss‚2.3772854962769676 1loss/SRqqt3/detection_loss‚1.5331480916132008 1loss/FBt3/assignment_loss‚0.05938629060983658 .loss/FBt1/detection_loss‚2.659167660607232 &loss/total_loss‚45.352908876716825 /loss/FRt4/detection_loss‚1.7245699238652985 % _timestamp‚1.7884243548704882e+09 /loss/FRt1/detection_loss‚2.2090442019204315 0loss/FRt1/assignment_loss‚3.5369970413545766 0loss/FRt4/assignment_loss‚1.2862131187209376 2loss/SRqqt4/assignment_loss‚0.8731510745482924 1loss/FBt1/assignment_loss‚0.27102344400330164 .loss/FRt3/detection_loss‚2.396717036658145 1loss/SRqqt4/detection_loss‚1.0553709807657246 /loss/FRt3/assignment_loss‚2.194960209917515 1loss/FBt2/assignment_loss‚0.29760077471170615 /loss/FRt2/detection_loss‚2.6818224608898165 trainer/global_step‚9799 _runtime‚7854.607949895 _step‚807 2loss/SRqqt2/assignment_loss‚1.5465268876184435 /loss/FBt4/detection_loss‚0.6073274013714568 0loss/FRt2/assignment_loss‚3.4703761835892997 0loss/FBt4/assignment_loss‚0.1291072370553002 .loss/FBt2/detection_loss‚5.364065905412038 .loss/FBt3/detection_loss‚4.271941661834717 2loss/SRqqt1/assignment_loss‚1.6471176510186571 2loss/SRqqt3/assignment_loss‚1.1544700172778808 1loss/SRqqt1/detection_loss‚2.0055181250760428 epoch‚8m±nÙc‚%Ý 1loss/FBt3/assignment_loss‚0.05938629060983658 .loss/FBt1/detection_loss‚2.659167660607232 &loss/total_loss‚45.352908876716825 /loss/FRt4/detection_loss‚1.7245699238652985 % _timestamp‚1.7884243548704882e+09 /loss/FRt1/detection_loss‚2.2090442019204315 0loss/FRt1/assignment_loss‚3.5369970413545766 0loss/FRt4/assignment_loss‚1.2862131187209376 2loss/SRqqt4/assignment_loss‚0.8731510745482924 1loss/FBt1/assignment_loss‚0.27102344400330164 .loss/FRt3/detection_loss‚2.396717036658145 1loss/SRqqt4/detection_loss‚1.0553709807657246 /loss/FRt3/assignment_loss‚2.194960209917515 1loss/FBt2/assignment_loss‚0.29760077471170615 /loss/FRt2/detection_loss‚2.6818224608898165 trainer/global_step‚9799 _runtime‚7854.607949895 _step‚807 2loss/SRqqt2/assignment_loss‚1.5465268876184435 /loss/FBt4/detection_loss‚0.6073274013714568 0loss/FRt2/assignment_loss‚3.4703761835892997 0loss/FBt4/assignment_loss‚0.1291072370553002 .loss/FBt2/detection_loss‚5.364065905412038 .loss/FBt3/detection_loss‚4.271941661834717 2loss/SRqqt1/assignment_loss‚1.6471176510186571 2loss/SRqqt3/assignment_loss‚1.1544700172778808 1loss/SRqqt1/detection_loss‚2.0055181250760428 epoch‚8 1loss/SRqqt2/detection_loss‚2.3772854962769676 1loss/SRqqt3/detection_loss‚1.5331480916132008§0`ôƒ%‚(:’ ¨áäÔ¹ÙË gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.748799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$li„%‚(:Þ ¨áäÔü±§€ disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375' proc.memory.availableMB‚ 1427.015625" proc.memory.rssMB‚ 2772.3984375 disk.rbd1.in‚0% disk.nvme0c0n1.out‚33899.19921875 disk.sda3.out‚ 427.8828125 disk.drbd2265.out‚0* proc.memory.percent‚16.921377182006836 disk.drbd2080.out‚0 disk.dm-0.out‚33899.19921875 disk.sda1.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.sda.out‚ 427.8828125 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 cpu‚25.357243862650943 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 427.8828125 disk.sda.in‚ 126.77734375# disk.nvme0n1.out‚33899.19921875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.3643203e+07 network.recv‚ 2.879606e+06) disk./.usagePercent‚17.47002965878005 disk.rbd1.out‚ 3.986328125$ memory_percent‚91.29018783569336% disk./.usageGB‚260.31606674194336 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0m¶Tœ…%‚(:‘ ·áäÔúÜ‚~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.87' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.974000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*^ÓRu†%‚(:ê ·áäÔ¨¿„€$ disk./.usageGB‚260.3327865600586 disk.sda3.out‚ 428.30859375 disk.sda.in‚ 126.77734375 disk.sda.out‚ 428.30859375& disk.nvme0n1.out‚33944.27490234375" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.3751319e+07 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚33944.27490234375 disk.drbd2080.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚33944.27490234375 disk.dm-1.out‚ 428.30859375 cpu‚25.356662386574413 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 3.986328125 disk.drbd1716.in‚0' proc.memory.availableMB‚ 1426.765625* proc.memory.percent‚16.921377182006836 network.recv‚ 2.884985e+06 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17* disk./.usagePercent‚17.471151739802625 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0$ memory_percent‚91.29171371459961 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875¡]œõ‡%‚(:’ ÆáäÔ£­~ gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.326' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.865199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oɪZtˆ%‚(:é ÆáäÔ‰û½€ disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 429.04296875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.out‚0# disk.dm-0.out‚33986.99755859375 disk.rbd1.out‚ 3.986328125 disk.sda.in‚ 126.77734375 disk.sda.out‚ 429.04296875 disk.drbd2266.out‚0# memory_percent‚94.1875696182251" proc.memory.rssMB‚ 2772.3984375 network.recv‚ 2.889658e+06* disk./.usagePercent‚17.472463010393007 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875& disk.nvme0n1.out‚33986.99755859375 network.sent‚ 2.3757375e+07 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0( proc.memory.availableMB‚ 952.30859375 disk.drbd2265.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.921377182006836$ disk./.usageGB‚260.3523254394531( disk.nvme0c0n1.out‚33986.99755859375 disk.sda3.out‚ 429.04296875 disk.drbd2080.in‚0 disk.sda1.in‚0 cpu‚25.35607829355173 proc.cpu.threads‚17¿K2‰%- _wandbruntime‚7890  _runtime‚7890‘ü¨ÉŠ%à trainer/global_step‚9849 % _timestamp‚1.7884243908546445e+09 _runtime‚7890.592170318 _step‚808 ' lr-AdamW/pg1‚1.7047549929885895e-05 ' lr-AdamW/pg2‚1.7047549929885895e-05 °ON΋%È ' lr-AdamW/pg1‚1.7047549929885895e-05 ' lr-AdamW/pg2‚1.7047549929885895e-05 trainer/global_step‚9849 % _timestamp‚1.7884243908546445e+09 _runtime‚7890.592170318 _step‚808¨#ƒ®”2Œ%- _wandbruntime‚7890  _runtime‚7890nsâ ç'%áO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8663065851904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8258112591872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186142400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626369767047680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958748 % _timestamp‚1.7884243908553338e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚809 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633782 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7890.593077921 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032856839680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112591872ìÌRlì'Ž%æO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8258768954368 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130954436 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130953985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83738516 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47215918 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8662891742208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚566837801435864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143511420 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186122362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚558609424186368 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186142400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8258112591872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186122362 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186142400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚581914774242816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚626369767047680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143511420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9849 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚566837129958748 % _timestamp‚1.7884243908553338e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329653822 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚449 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚626369790987776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 _step‚809 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚573656679653888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143511422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚581914774242816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329633782 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8228360210136 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7890.593077921 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚451 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚635032856839680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚558609441225728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8258112591872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143511422 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8663065851904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚573656661650944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83738518 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932019712 OGDeviceStatsMonitor.on_train_batch_end/inactive_spl—è|•à!it.small_pool.current‚451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130967046 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130966593 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47220360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626431441725440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186160880 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8259588395520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47219909 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8663752055296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚7891.294085505 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946709504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚810 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186160880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635095169840640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9849 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚626431417785344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8663752055296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186140816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 329666252 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _runtime‚7891.294085505 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581971946709504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚573713032692736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚810 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186160880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329686318 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚635095169840640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚558664345356288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 329686318 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚83746686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚573713014689792 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143525438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8258932019712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143525436 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130967046 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130966593 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚566892850524272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47220360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚566893522001596 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚626431441725440ªxפ*2’%- _wandbruntime‚7891  _runtime‚7891Û³AZ“%Ô 2loss/SRqqt1/assignment_loss‚1.6997222654197528 1loss/SRqqt4/assignment_loss‚1.070579672260071 _runtime‚7891.295671928 /loss/FRt3/assignment_loss‚2.352958033693598 1loss/SRqqt3/assignment_loss‚1.306066959603306 0loss/FBt2/assignment_loss‚0.2720609181560576 1loss/FBt4/assignment_loss‚0.21415643377494623 /loss/FRt4/detection_loss‚1.7940449949486978 0loss/SRqqt1/detection_loss‚1.963646601723588 1loss/SRqqt3/detection_loss‚1.6311512219116968 0loss/FBt4/detection_loss‚0.39550927131265207 0loss/FRt1/assignment_loss‚3.3107381247459573 /loss/FBt2/detection_loss‚2.8116894165674844 % _timestamp‚1.7884243915582004e+09 /loss/FRt2/detection_loss‚2.5707876205444338 .loss/FRt3/detection_loss‚2.304992217332759 1loss/SRqqt2/detection_loss‚2.4272819250822066 /loss/FBt3/detection_loss‚2.2131405572096505 .loss/FRt2/assignment_loss‚3.72314688762029 1loss/FBt3/assignment_loss‚0.42014321063955623 /loss/FRt1/detection_loss‚2.1260895500791834 _step‚811 2loss/SRqqt2/assignment_loss‚1.8528163378437361 0loss/FBt1/assignment_loss‚0.7461618619660536 epoch‚8 trainer/global_step‚9849 0loss/FRt4/assignment_loss‚1.5885690269725663 .loss/FBt1/detection_loss‚2.481150726477305 &loss/total_loss‚42.492014346603035 0loss/SRqqt4/detection_loss‚1.215410510717488ÐVê_”%Ù 2loss/SRqqt1/assignment_loss‚1.6997222654197528 1loss/SRqqt4/assignment_loss‚1.070579672260071 _runtime‚7891.295671928 /loss/FRt3/assignment_loss‚2.352958033693598 1loss/SRqqt3/assignment_loss‚1.306066959603306 0loss/FBt2/assignment_loss‚0.2720609181560576 1loss/FBt4/assignment_loss‚0.21415643377494623 /loss/FRt4/detection_loss‚1.7940449949486978 0loss/SRqqt1/detection_loss‚1.963646601723588 1loss/SRqqt3/detection_loss‚1.6311512219116968 0loss/FBt4/detection_loss‚0.39550927131265207 0loss/FRt1/assignment_loss‚3.3107381247459573 /loss/FBt2/detection_loss‚2.8116894165674844 % _timestamp‚1.7884243915582004e+09 /loss/FRt2/detection_loss‚2.5707876205444338 .loss/FRt3/detection_loss‚2.304992217332759 1loss/SRqqt2/detection_loss‚2.4272819250822066 /loss/FBt3/detection_loss‚2.2131405572096505 .loss/FRt2/assignment_loss‚3.72314688762029 1loss/FBt3/assignment_loss‚0.42014321063955623 /loss/FRt1/detection_loss‚2.1260895500791834 _step‚811 2loss/SRqqt2/assignment_loss‚1.8528163378437361 0loss/FBt1/assignment_loss‚0.7461618619660536 epoch‚8 trainer/global_step‚9849 0loss/FRt4/assignment_loss‚1.5885690269725663 .loss/FBt1/detection_loss‚2.481150726477305 &loss/total_loss‚42.492014346603035 0loss/SRqqt4/detection_loss‚1.215410510717488«SË`㜕%‚(:‘ ÕáäÔáÅž~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.372' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.27440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž6vûr–%‚(:ç ÕáäÔµò‘ network.recv‚ 2.893542e+06$ disk./.usageGB‚260.3614692687988 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.6875 cpu‚25.355495122037865) disk./.usagePercent‚17.47307666045258 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 429.39453125 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚34051.51904296875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚34051.51904296875% proc.memory.availableMB‚ 952.28125 network.sent‚ 2.3763279e+07 disk.drbd1716.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.dm-1.out‚ 429.39453125 disk.rbd1.in‚0 disk.rbd1.out‚ 3.986328125 disk.nvme0c0n1.in‚4.6875$ memory_percent‚94.18773651123047& disk.nvme0n1.out‚34051.51904296875 disk.sda3.out‚ 429.39453125 disk.drbd2080.in‚0 disk.drbd2080.out‚0ÛЯܒ—%‚(:‡ äáäÔ™§™~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚160.256' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0512 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åG¢2l˜%‚(:á äáäÔëýù" proc.memory.rssMB‚ 2772.3984375) disk./.usagePercent‚17.47366778184914 disk.drbd2266.out‚0 disk.sda3.out‚ 429.92578125% disk./.usageGB‚260.37027740478516 disk.sda1.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921377182006836 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.986328125 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0$ memory_percent‚94.19076442718506 disk.rbd0.out‚0% disk.nvme0c0n1.out‚34100.24609375 disk.dm-1.out‚ 429.92578125 disk.sda.out‚ 429.92578125 disk.sda1.in‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 cpu‚25.354926270438902 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.out‚34100.24609375( proc.memory.availableMB‚ 951.78515625 network.sent‚ 2.3872236e+07 network.recv‚ 2.900008e+06 disk.drbd1716.out‚0# disk.nvme0n1.out‚34100.24609375 disk.drbd2265.in‚0[©h2™%- _wandbruntime‚7927  _runtime‚7927{AÉßÈš% ' lr-AdamW/pg1‚1.6320601906290366e-05 ' lr-AdamW/pg2‚1.6320601906290366e-05 trainer/global_step‚9899 $ _timestamp‚1.788424427628274e+09 _runtime‚7927.365795489 _step‚8129|…Í›%Ç _step‚812 ' lr-AdamW/pg1‚1.6320601906290366e-05 ' lr-AdamW/pg2‚1.6320601906290366e-05 trainer/global_step‚9899 $ _timestamp‚1.788424427628274e+09 _runtime‚7927.365795489¬Ç0ÃØ2œ%- _wandbruntime‚7927  _runtime‚7927Ú£hQÙ%àO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576474313593344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8299098397696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638158136915968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561354630714368 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8299754760192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187068452 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144212222 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚584773411991040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269195549544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47440465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584774086356480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84146918 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576474331596288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826156 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131586943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754760192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9899 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187048414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111788 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098397696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47440027 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚813 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7927.366735462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638158318188544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303272 F@DeviceStatsMonitor.on_train_batch_start/resÇßµ erved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788424427629007e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144212220 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84146916 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629452327870976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131587383 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚584773411991040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576474331596288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚569623171826156 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131586943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8299754760192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚561354647753728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 trainer/global_step‚9899 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187048414 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8268541111788 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8299098397696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576474313593344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47440027 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚813 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚7927.366735462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚576474331596288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638158318188544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569623843303272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 $ _timestamp‚1.788424427629007e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187068452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144212220 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84146916 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629452327870976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131587383 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269195549544 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144212220 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47440465 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚584774086356480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84146918 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706696491008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884244283204083e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187066868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9899 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576530684635136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629513954668544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144226238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638220836613120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258836480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569679563869004 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258836480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47444518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270011984716 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8300574201344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131599602 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚814 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131600061 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 7928.05839495 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569678892391680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220651159552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576530684635136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚584831258836480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47444518 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270011984716 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8300574201344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131599602 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚584830584457728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144226236 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _step‚814 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331293104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131600061 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187066868 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _runtime‚ 7928.05839495 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8299917825536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8299917825536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569678892391680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84155084 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638220651159552 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚576530684635136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8300574201344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8706696491008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 % _timestamp‚1.7884244283204083e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187086932 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187066868 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚561409551884288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚9899 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144226236 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84155086 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144226238 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331313170 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚576530684635136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629513954668544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚561409534844928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚576530666632192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 331293104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144226238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚638220836613120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚584831258836480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569679563869004 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974466 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84555318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚564099854281728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8748999586816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748825477120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8309376451196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _runtime‚7964.329751042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144913020 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632534840814080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332887486 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187994504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132220062 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47664746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47665183 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8340740566016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8310030888952 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587632724104704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740566016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚572409213693564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283666291200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144913020 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579291983538688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884244645920136e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587632049739264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8340084203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632534864754176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907526 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0^F´ì'®%æO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144913020 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚579291965535744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632534840814080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332887486 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚579291983538688 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187994504 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132220062 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47664746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47665183 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8340740566016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8310030888952 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚587632724104704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8340740566016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144913022 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚572409213693564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144913022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84555316 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641283666291200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144913020 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚579291983538688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7884244645920136e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚587632049739264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8340084203520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚437 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632534864754176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚572409885170680 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187994504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 332907526 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174109696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 198049792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚587632049739264 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187974466 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84555318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚564099854281728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187974466 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8748999586816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8748825477120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8309376451196 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _runtime‚7964.329751042 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2±çNÖÙ’¯%‚(:‡ ‘âäÔçÁ¦~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 "²°o°%‚(:ä ‘âäÔɧ­€ network.recv‚ 2.913923e+06 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.sent‚ 2.3992098e+07" disk.dm-0.out‚34266.1943359375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 3.986328125' disk.nvme0c0n1.out‚34266.1943359375 disk.sda2.out‚0 disk.dm-1.out‚ 431.5234375 disk.sda.out‚ 431.5234375 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 431.5234375 cpu‚25.353175200317562) disk./.usagePercent‚17.47679696194016 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.sda2.in‚0$ memory_percent‚94.18818950653076( proc.memory.availableMB‚ 952.20703125$ disk./.usageGB‚260.4169044494629 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.dm-1.in‚ 126.77734375% disk.nvme0n1.out‚34266.1943359375 disk.rbd0.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921377182006836á¬Ô2±%- _wandbruntime‚7965  _runtime‚79655ÇX]²%êM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8749676848640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚579348318574592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8341560007168 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84563486 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚564154758412288 $ _timestamp‚1.788424465287514e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332940022 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188012984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144927038 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚564154741372928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132233128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 trainer/global_step‚9949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47669192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8310847324124 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7965.025416974 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144927038 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749848847872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560007168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340903631360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903631360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132232676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641346168400384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47669642 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚818 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736412 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132232676 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚641346168400384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144927036 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47669642 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187992920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚579348318574592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚572464934259088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _step‚818 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚572465605736412 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚579348336577536 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144927036 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632596491551744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132233128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚632596515491840 trainer/global_step‚9949 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47669192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8310847324124 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8310192886160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332940022 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7965.025416974 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144927038 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8749848847872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8341560007168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84563484 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8340903631360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340903631360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332919956 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚587689896584704 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188920556 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47889856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84963718 _runtime‚8001.035059494 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791680266752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132854014 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47890296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644409247625216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605óù_»+À%æO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pŒ¸+ÒÁ&ool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8381726371840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8381070009344 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132854014 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47890296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚644409247625216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84963716 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _step‚821 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8350211790604 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚590491361852928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145613820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334514338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚644409057964032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188920556 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132853572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚575195255560972 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚582109635481088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚566845060809728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚442 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8381726371840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566845043770368 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8381070009344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚582109617478144 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145613820 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚9999 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188900518 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 % _timestamp‚1.7884245012972054e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590490687487488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 188920556 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚635617377697280 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8350866228360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590490687487488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145613822 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚575195927038088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582109617478144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚582109635481088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47889856 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚635617401637376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188900518 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84963718 _runtime‚8001.035059494 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145613822 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334534378 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8791680266752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84971884 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792572035072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚9999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8382545812992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132866263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8001.728174051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168676352 _step‚822 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8351682663532O¡G—ö&Ã%ðM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792572035072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚575251647603820 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚9999 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8382545812992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132866263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚8001.728174051 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84971886 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚575250976126496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566899947900928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8351028225568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188918972 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188918972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚635679028434944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚566899964940288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 168676352 _step‚822 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334566874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8351682663532 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145627838 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590547859954176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84971884 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145627836 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚582165988519936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334566874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360¶œÚ2Ä%- _wandbruntime‚8001  _runtime‚8001WJ«ßWÅ%Ñ 1loss/SRqqt1/detection_loss‚2.0657638729886805 .loss/FBt1/detection_loss‚2.437577724456787 % _timestamp‚1.7884245019917238e+09 /loss/FRt3/assignment_loss‚2.715909805893898 1loss/FBt3/assignment_loss‚0.08020384654325123 2loss/SRqqt3/assignment_loss‚1.2795118535203593 /loss/FBt4/detection_loss‚0.5210812329258545 epoch‚8 .loss/FBt2/detection_loss‚2.085698090493679 %loss/total_loss‚42.65802858495438 trainer/global_step‚9999 .loss/FRt2/detection_loss‚2.986306925614675 .loss/FRt4/detection_loss‚1.631727811608774 1loss/SRqqt2/detection_loss‚2.3157680736434076 /loss/FRt1/assignment_loss‚3.538993375168906 2loss/SRqqt4/assignment_loss‚0.8232812744968417 0loss/FBt2/assignment_loss‚0.1372911391663365 1loss/SRqqt4/detection_loss‚0.9095601509811718 _step‚823 1loss/FBt4/assignment_loss‚0.10596029398505002 /loss/FRt2/assignment_loss‚4.188743123301753 .loss/FRt1/detection_loss‚2.361217172609435 .loss/FRt3/detection_loss‚2.605889054387808 .loss/FBt3/detection_loss‚3.336216906706492 _runtime‚8001.729147888 0loss/FRt4/assignment_loss‚1.3519638068974018 2loss/SRqqt1/assignment_loss‚1.3056932568074542 2loss/SRqqt2/assignment_loss‚1.7119215181277645 0loss/FBt1/assignment_loss‚0.2741788641682693 1loss/SRqqt3/detection_loss‚1.8875694104603358‹Q\Æ%Ö /loss/FRt3/assignment_loss‚2.715909805893898 1loss/FBt3/assignment_loss‚0.08020384654325123 2loss/SRqqt3/assignment_loss‚1.2795118535203593 /loss/FBt4/detection_loss‚0.5210812329258545 epoch‚8 .loss/FBt2/detection_loss‚2.085698090493679 %loss/total_loss‚42.65802858495438 trainer/global_step‚9999 .loss/FRt2/detection_loss‚2.986306925614675 .loss/FRt4/detection_loss‚1.631727811608774 1loss/SRqqt2/detection_loss‚2.3157680736434076 /loss/FRt1/assignment_loss‚3.538993375168906 2loss/SRqqt4/assignment_loss‚0.8232812744968417 0loss/FBt2/assignment_loss‚0.1372911391663365 1loss/SRqqt4/detection_loss‚0.9095601509811718 _step‚823 1loss/FBt4/assignment_loss‚0.10596029398505002 /loss/FRt2/assignment_loss‚4.188743123301753 .loss/FRt1/detection_loss‚2.361217172609435 .loss/FRt3/detection_loss‚2.605889054387808 .loss/FBt3/detection_loss‚3.336216906706492 _runtime‚8001.729147888 0loss/FRt4/assignment_loss‚1.3519638068974018 2loss/SRqqt1/assignment_loss‚1.3056932568074542 2loss/SRqqt2/assignment_loss‚1.7119215181277645 0loss/FBt1/assignment_loss‚0.2741788641682693 1loss/SRqqt3/detection_loss‚1.8875694104603358 1loss/SRqqt1/detection_loss‚2.0657638729886805 .loss/FBt1/detection_loss‚2.437577724456787 % _timestamp‚1.7884245019917238e+09·éœ Ç%‚(:ÿç3õˆ† ¾âäÔ©¡¢~ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.955' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.991 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'*wÈ%‚(:ì ¾âäÔîÕ‚ network.recv‚ 2.930243e+06 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚34417.08935546875 network.sent‚ 2.4193309e+07 disk.rbd1.out‚ 4.072265625& disk.nvme0n1.out‚34417.08935546875 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0$ memory_percent‚94.19004917144775% disk./.usageGB‚260.42831802368164 disk.drbd2265.in‚0 disk.rbd1.in‚0 cpu‚25.351473220942125 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 433.60546875 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚34417.08935546875 disk.drbd2266.in‚0 disk.sda.out‚ 433.60546875 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 433.60546875" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875* disk./.usagePercent‚17.477562936482606 disk.rbd0.in‚0( proc.memory.availableMB‚ 951.90234375* proc.memory.percent‚16.921377182006836 ñûL‘É%‚(:† ÍâäÔþÕ~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É nÊ%‚(:ã ÍâäÔÿŒû network.sent‚ 2.4301749e+07 network.recv‚ 2.936065e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 4.072265625 disk.drbd1716.in‚0# memory_percent‚94.1887378692627' proc.memory.availableMB‚ 952.1171875 disk.dm-1.in‚ 126.77734375 disk.nvme0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚34462.7119140625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 434.0078125 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375 cpu‚25.350918446130258 proc.cpu.threads‚17* disk./.usagePercent‚17.479090277431883 disk.dm-0.in‚4.6875% disk./.usageGB‚260.45107650756836 disk.sda2.out‚0 disk.sda3.out‚ 434.0078125% disk.nvme0n1.out‚34462.7119140625 disk.rbd0.out‚0' disk.nvme0c0n1.out‚34462.7119140625 disk.dm-1.out‚ 434.0078125 disk.sda2.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.9213771820068368Vb\2Ë%- _wandbruntime‚8037  _runtime‚8037ÕY%ÊÌ%Ä _step‚824 ' lr-AdamW/pg1‚1.4225403987161271e-05 ' lr-AdamW/pg2‚1.4225403987161271e-05 trainer/global_step‚10049 % _timestamp‚1.7884245378450713e+09 _runtime‚8037.582574552j3aÈÏÍ%É _runtime‚8037.582574552 _step‚824 ' lr-AdamW/pg1‚1.4225403987161271e-05 ' lr-AdamW/pg2‚1.4225403987161271e-05 trainer/global_step‚10049 % _timestamp‚1.7884245378450713e+09¸¤ÏɃ2Î%- _wandbruntime‚8037  _runtime‚8037²}‡Pç'Ï%áO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146314620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584927287423488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133487869 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8037.583387696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055815168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48115753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999601152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834554857984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189826570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133487869 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146314622 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 _runtime‚8037.583387696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8422055815168 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8391047130012 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48115753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8391701567768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚593349999601152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8834554857984 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584927269420544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569590250298368 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146314620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189826570 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85372118 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85372116 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚593349325235712 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 _runtime‚8038.282161796 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193726 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638761565318144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875243008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚569645171468288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚593406497702400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133500537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚593407172081152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189845024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647597169671168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚826 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133500997 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;5Device§‰·tFStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48120711 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240m]¼N÷&Ó%ñM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚647597169671168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚584983622459392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚578037017993904 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚826 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133500997 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391863564976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48120711 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578037689471228 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336173660 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189865088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚593407172081152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638761589258240 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 _runtime‚8038.282161796 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 336193726 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422875243008 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638761565318144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422875243008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚458 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚569645171468288 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚593406497702400 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133500537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚593407172081152 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146328638 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚584983622459392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189845024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189845024 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336193726 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85380286 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414ºKp“2Ô%- _wandbruntime‚8038  _runtime‚8038^½âxYÕ%Ó /loss/FRt2/assignment_loss‚3.128693260929801 2loss/SRqqt3/assignment_loss‚0.9928417453528241 1loss/FBt1/assignment_loss‚0.14654971858752625 /loss/FRt1/detection_loss‚2.1134787698586783 /loss/FRt3/detection_loss‚1.9123774455352263 /loss/FBt1/detection_loss‚2.3982453686850413 0loss/FRt1/assignment_loss‚3.5749081139471017 /loss/FRt3/assignment_loss‚2.101601521535353 1loss/SRqqt2/assignment_loss‚1.560317249634327 2loss/SRqqt4/assignment_loss‚0.8965189556210217 /loss/FBt3/detection_loss‚2.2877677202224733 _runtime‚ 8038.28321874 0loss/SRqqt2/detection_loss‚2.070140010271317 1loss/SRqqt4/detection_loss‚1.0953760519623756 %loss/total_loss‚38.81045646213077 epoch‚8 /loss/FBt2/assignment_loss‚0.387051573197823 1loss/FBt3/assignment_loss‚0.31412829116452484 1loss/FBt4/assignment_loss‚0.10790251193633221 /loss/FRt2/detection_loss‚3.3933477212082255 % _timestamp‚1.7884245385458047e+09 _step‚827 trainer/global_step‚10049 .loss/FRt4/detection_loss‚1.469564160587741 0loss/SRqqt3/detection_loss‚1.512112897453886 /loss/FBt2/detection_loss‚2.4748407900333405 0loss/FRt4/assignment_loss‚1.2397162248106564 1loss/SRqqt1/assignment_loss‚1.232323231504244 0loss/SRqqt1/detection_loss‚1.814266579992631 /loss/FBt4/detection_loss‚0.5863865480983051ˆÍ^Ö%Ø .loss/FRt4/detection_loss‚1.469564160587741 0loss/SRqqt3/detection_loss‚1.512112897453886 /loss/FBt2/detection_loss‚2.4748407900333405 0loss/FRt4/assignment_loss‚1.2397162248106564 1loss/SRqqt1/assignment_loss‚1.232323231504244 0loss/SRqqt1/detection_loss‚1.814266579992631 /loss/FBt4/detection_loss‚0.5863865480983051 /loss/FRt2/assignment_loss‚3.128693260929801 2loss/SRqqt3/assignment_loss‚0.9928417453528241 1loss/FBt1/assignment_loss‚0.14654971858752625 /loss/FRt1/detection_loss‚2.1134787698586783 /loss/FRt3/detection_loss‚1.9123774455352263 /loss/FBt1/detection_loss‚2.3982453686850413 0loss/FRt1/assignment_loss‚3.5749081139471017 /loss/FRt3/assignment_loss‚2.101601521535353 1loss/SRqqt2/assignment_loss‚1.560317249634327 2loss/SRqqt4/assignment_loss‚0.8965189556210217 /loss/FBt3/detection_loss‚2.2877677202224733 _runtime‚ 8038.28321874 0loss/SRqqt2/detection_loss‚2.070140010271317 1loss/SRqqt4/detection_loss‚1.0953760519623756 %loss/total_loss‚38.81045646213077 epoch‚8 /loss/FBt2/assignment_loss‚0.387051573197823 1loss/FBt3/assignment_loss‚0.31412829116452484 1loss/FBt4/assignment_loss‚0.10790251193633221 /loss/FRt2/detection_loss‚3.3933477212082255 % _timestamp‚1.7884245385458047e+09 _step‚827 trainer/global_step‚10049»MšÑ’×%‚(:‡ ÜâäÔ“¡±~ gpu.0.gpu‚37.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.8964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VwùiØ%‚(:Þ ÜâäÔáÃÇ€ network.recv‚ 2.940969e+06$ disk./.usageGB‚260.4584045410156 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375' proc.memory.availableMB‚ 952.1015625 proc.cpu.threads‚17 disk.dm-1.out‚ 434.69140625 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.6875 network.sent‚ 2.4308194e+07) disk./.usagePercent‚17.47958206790516 disk.drbd2080.out‚0 disk.drbd1716.out‚0$ memory_percent‚94.18883323669434 disk.dm-0.out‚34493.77734375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚34493.77734375 disk.sda2.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0# disk.nvme0n1.out‚34493.77734375 disk.rbd0.in‚0 disk.sda3.out‚ 434.69140625* proc.memory.percent‚16.921377182006836 disk.sda.out‚ 434.69140625 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.134765625 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2772.3984375 cpu‚25.3503549221841Ң̘Ù%‚(:’ ëâäÔÂÄ–~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ósžãgÚ%‚(:Ü ëâäÔùå— disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 435.21875 network.sent‚ 2.441624e+07 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2266.in‚0$ memory_percent‚94.18895244598389( proc.memory.availableMB‚ 952.08203125 cpu‚25.34979335408818 network.recv‚ 2.94657e+06 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375* disk./.usagePercent‚17.480068482220215 disk.sda1.in‚0% disk.nvme0n1.out‚34548.6904296875' disk.nvme0c0n1.out‚34548.6904296875 disk.sda3.out‚ 435.21875 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.134765625 disk.sda.out‚ 435.21875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0" disk.dm-0.out‚34548.6904296875 disk.sda.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.sda2.in‚0* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17$ disk./.usageGB‚260.4656524658203 disk.sda1.out‚0\ò’Û%‚(:‡ úâäÔŒÙÌ~ gpu.0.gpu‚53.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚128.362' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6724 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸~#6oÜ%‚(:ä úâäÔúÅ·€ disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875& disk.nvme0n1.out‚34595.36572265625 disk.drbd1716.out‚0% disk./.usageGB‚260.48628997802734 disk.sda3.out‚ 435.90625 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 435.90625 disk.sda.in‚ 126.77734375 cpu‚25.349242760035963 disk.rbd0.in‚0$ memory_percent‚94.18747425079346 network.sent‚ 2.4422188e+07 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 435.90625( disk.nvme0c0n1.out‚34595.36572265625( proc.memory.availableMB‚ 952.32421875" proc.memory.rssMB‚ 2772.3984375 disk.drbd1716.in‚0 network.recv‚ 2.950922e+06* disk./.usagePercent‚17.481453482980456 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 4.1669921875 disk.nvme0c0n1.in‚4.6875* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17# disk.dm-0.out‚34595.36572265625 disk.sda1.in‚0b¹Ú2Ý%- _wandbruntime‚8074  _runtime‚8074}œbGÊÞ%Ä ' lr-AdamW/pg1‚1.3556038552603352e-05 ' lr-AdamW/pg2‚1.3556038552603352e-05 trainer/global_step‚10099 % _timestamp‚1.7884245745757234e+09 _runtime‚8074.313275512 _step‚828…¼D…Ïß%É trainer/global_step‚10099 % _timestamp‚1.7884245745757234e+09 _runtime‚8074.313275512 _step‚828 ' lr-AdamW/pg1‚1.3556038552603352e-05 ' lr-AdamW/pg2‚1.3556038552603352e-05¼&/'2à%- _wandbruntime‚8074  _runtime‚8074ª©ÛEè'á%âO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134122664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697983488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚587744921362944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190772660 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147015420 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8074.314476428 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190752622 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641782451463680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚587744921362944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8431882469420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650660162624000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641782475403776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8463041620992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8432536907176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48342146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884245745766184e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8877687220224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596208637349376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85780516 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚596208637349376 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8463697983488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚587744921362944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190772660 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚580767339295788 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147015420 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚580768010772904 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚8074.314476428 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190752622 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚641782451463680 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚587744921362944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147015422 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8431882469420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚587744939365888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650660162624000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚641782475403776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚596207962983936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8463041620992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8432536907176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48342146 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884245745766184e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944½‚SÞE2ã%- _wandbruntime‚8075  _runtime‚8075›ç¶ò&ä%ìM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650722495405568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147029436 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚587801274401792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚641844126141440 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚580823059861312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338636 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641844102201344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48346093 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134134777 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8075.007187157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722680859136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48346541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596265809829376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878554717696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147029438 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147029436 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚580823059861312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147029438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8464517424640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚580823731338636 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚641844102201344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85788684 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48346093 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134134777 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8075.007187157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650722680859136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8433353342348 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48346541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚596265809829376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚587801292404736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337820578 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337820578 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8878554717696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚587801274401792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¾Êuš2æ%- _wandbruntime‚8075  _runtime‚8075k\æ7Xç%Ò 0loss/FRt3/assignment_loss‚2.1750210858881474 2loss/SRqqt3/assignment_loss‚1.1942943301180313 0loss/SRqqt2/detection_loss‚2.584731767574946 .loss/FBt4/detection_loss‚0.697188465496404 0loss/SRqqt4/detection_loss‚1.054627502364093 .loss/FBt1/detection_loss‚3.284158850709597 -loss/FBt3/detection_loss‚3.97089421749115 %loss/total_loss‚43.61687281597383 epoch‚8 .loss/FRt2/detection_loss‚3.452871912717819 2loss/SRqqt2/assignment_loss‚1.5632983517867547 0loss/FBt2/assignment_loss‚0.1744973342073182 1loss/FBt4/assignment_loss‚0.12812070986481944 /loss/FRt3/detection_loss‚2.2155972871308527 1loss/SRqqt3/detection_loss‚1.8556187715509842 0loss/FRt2/assignment_loss‚3.6236539840698243 /loss/FRt4/assignment_loss‚1.058643012046814 0loss/FBt3/assignment_loss‚0.3065155786462128 /loss/FBt2/detection_loss‚2.2299956017070346 _step‚831 2loss/SRqqt4/assignment_loss‚1.0315823684433016 /loss/FRt4/detection_loss‚1.6241418703645467 % _timestamp‚1.7884245752707765e+09 0loss/FRt1/assignment_loss‚3.6844288445100553 0loss/FBt1/assignment_loss‚0.3463282824183504 _runtime‚8075.008210337 2loss/SRqqt1/assignment_loss‚1.1726290669647002 /loss/FRt1/detection_loss‚2.3029478381319746 0loss/SRqqt1/detection_loss‚1.885085781770093 trainer/global_step‚10099¤%Ä ]è%× _runtime‚8075.008210337 2loss/SRqqt1/assignment_loss‚1.1726290669647002 /loss/FRt1/detection_loss‚2.3029478381319746 0loss/SRqqt1/detection_loss‚1.885085781770093 trainer/global_step‚10099 0loss/FRt3/assignment_loss‚2.1750210858881474 2loss/SRqqt3/assignment_loss‚1.1942943301180313 0loss/SRqqt2/detection_loss‚2.584731767574946 .loss/FBt4/detection_loss‚0.697188465496404 0loss/SRqqt4/detection_loss‚1.054627502364093 .loss/FBt1/detection_loss‚3.284158850709597 -loss/FBt3/detection_loss‚3.97089421749115 %loss/total_loss‚43.61687281597383 epoch‚8 .loss/FRt2/detection_loss‚3.452871912717819 2loss/SRqqt2/assignment_loss‚1.5632983517867547 0loss/FBt2/assignment_loss‚0.1744973342073182 1loss/FBt4/assignment_loss‚0.12812070986481944 /loss/FRt3/detection_loss‚2.2155972871308527 1loss/SRqqt3/detection_loss‚1.8556187715509842 0loss/FRt2/assignment_loss‚3.6236539840698243 /loss/FRt4/assignment_loss‚1.058643012046814 0loss/FBt3/assignment_loss‚0.3065155786462128 /loss/FBt2/detection_loss‚2.2299956017070346 _step‚831 2loss/SRqqt4/assignment_loss‚1.0315823684433016 /loss/FRt4/detection_loss‚1.6241418703645467 % _timestamp‚1.7884245752707765e+09 0loss/FRt1/assignment_loss‚3.6844288445100553 0loss/FBt1/assignment_loss‚0.3463282824183504¿È Åšé%‚(: ‰ãäÔ¡ÀŸ~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.760000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Uиpê%‚(:å ‰ãäÔºœñ€ network.sent‚ 2.442812e+07 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.1669921875& disk.nvme0n1.out‚34649.80517578125 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17% disk./.usageGB‚260.49778747558594 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375# disk.dm-0.out‚34649.80517578125 disk.dm-1.out‚ 436.390625 disk.sda3.in‚ 126.77734375 cpu‚25.34868665458703 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚ 2.954843e+06 disk.sda.out‚ 436.390625( disk.nvme0c0n1.out‚34649.80517578125 disk.sda2.in‚0 disk.drbd2080.out‚0( proc.memory.availableMB‚ 952.05078125 disk.sda.in‚ 126.77734375* disk./.usagePercent‚17.482225089688654 disk.drbd2266.out‚0 disk.sda3.out‚ 436.390625$ memory_percent‚94.18914318084717 disk.sda1.out‚0 disk.drbd2080.in‚0 BÞY’ë%‚(:‡ ˜ãäÔýìü} gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä¦ì%‚(:ß ˜ãäÔ÷Óæ€ network.recv‚ 2.961287e+06 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚34691.88330078125 proc.cpu.threads‚17 network.sent‚ 2.4537213e+07 disk.rbd1.out‚ 4.22998046875 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚34691.88330078125 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.out‚436.9375 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚436.9375% disk./.usageGB‚260.51321029663086 disk.sda1.out‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚34691.88330078125 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.dm-1.out‚436.9375" proc.memory.rssMB‚ 2772.3984375$ memory_percent‚9H–Ä4.18911933898926' proc.memory.availableMB‚ 952.0546875) disk./.usagePercent‚17.48326012814959 disk.drbd2265.out‚0 cpu‚25.34810490684938* proc.memory.percent‚16.921377182006836ƒSŽ2í%- _wandbruntime‚8111  _runtime‚8111Þ#+¸Êî%Ä % _timestamp‚1.7884246115478277e+09 _runtime‚8111.285380783 _step‚832 ' lr-AdamW/pg1‚1.2901425077939186e-05 ' lr-AdamW/pg2‚1.2901425077939186e-05 trainer/global_step‚10149Ø—'MÏï%É % _timestamp‚1.7884246115478277e+09 _runtime‚8111.285380783 _step‚832 ' lr-AdamW/pg1‚1.2901425077939186e-05 ' lr-AdamW/pg2‚1.2901425077939186e-05 trainer/global_step‚10149À|{õh2ð%- _wandbruntime‚8111  _runtime‚8111×SSè'ñ%âO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785780073472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚583553381163196 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275097600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884246115485291e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027426816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575080680393728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8472717808828 trainer/global_step‚10149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653785607189504 _step‚833 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48566967 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147716222 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86188918 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172883968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48567391 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚590562591308288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ô5Ðí'ò%çO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644864988346880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 148943872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚583554052640312 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147716220 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚599067275097600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147716220 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644865012286976 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339394894 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8504683789312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8504027426816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚575080680393728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8472717808828 trainer/global_step‚10149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚599066600732160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚653785607189504 _step‚833 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48566967 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8473372246584 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚575080663354368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86188916 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147716222 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339414934 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86188918 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172883968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚599066600732160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48567391 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚590562591308288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191678674 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191698712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚590562573305344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚590562573305344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚653785780073472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191678674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚583553381163196 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚599067275097600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 % _timestamp‚1.7884246115485291e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653848123595776 trainer/global_step‚10149 % _timestamp‚1.7884246122375374e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846854656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48571447 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191717192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206044 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503230464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921484511232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575135584524288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147730236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚409 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773198848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8474188681756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚411 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134768531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447577600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653848304855040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134768942 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414E8õ%ñM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339447430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191717192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191717192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86197084 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 157319168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚583609773206044 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8505503230464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚590618926344192 TEDevicŒ–³÷ eStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921484511232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚575135584524288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚644926663024640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147730236 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚409 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚599123773198848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8474188681756 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚411 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134768531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚599124447577600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339447430 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚653848304855040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339427364 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚599124447577600 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134768942 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590618944347136 _runtime‚8111.975747388 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191697128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590618944347136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147730238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 181259264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚653848123595776 trainer/global_step‚10149 % _timestamp‚1.7884246122375374e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8504846854656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48571447 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135390638 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656911119374848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8545669595136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148417020 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647947549170176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192604726 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48793788 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚601925238480384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8963594144768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 % _timestamp‚1.7884246483935874e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8545013232640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192604726 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192624764 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341041786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912845824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚837 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚10199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669595136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8148.131340382.óazÀ„&çO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8514207585992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192624764 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8545669595136 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148417020 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647947549170176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192604726 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚601925238480384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577825886921728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚593380225247744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48793788 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚601925238480384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚586340094507720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8963594144768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 % _timestamp‚1.7884246483935874e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8545013232640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 192604726 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192624764 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341041786 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚577825869882368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341021746 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148417022 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚601925912845824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment¼LD,-.large_pool.peak‚430 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚601925912845824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚593380243250688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86597316 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚837 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 trainer/global_step‚10199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚593380243250688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8545669595136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86597318 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚8148.131340382 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚647947525230080 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135391106 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚586339423030604 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148417020 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135390638 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚656911119374848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚601983085325824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135403317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48797833 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973620681216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86605486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 $ _timestamp‚1.788424649086979e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085325824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8546489036288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8546489036288 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚461 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648009199907840 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341074282 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341054216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚601983085325824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚577880774012928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚593436596289536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148431036 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135403317 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48797833 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚656973620681216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192643244 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86605486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚459 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577880791052288 $ _timestamp‚1.788424649086979e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚601983085325824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192643244 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ÆÇ)ƒ2ˆ&- _wandbruntime‚8148  _runtime‚8148m‚0V‰&Ð 2loss/SRqqt4/assignment_loss‚0.8740000726726765 0loss/FBt1/assignment_loss‚0.1297996485605836 trainer/global_step‚10199 1loss/FBt2/assignment_loss‚0.34888488723441924 1loss/FBt3/assignment_loss‚0.12270193605218083 0loss/SRqqt1/detection_loss‚1.701277063290278 _runtime‚8148.826164366 /loss/FRt2/detection_loss‚3.0185726664283057 /loss/FRt4/detection_loss‚1.6199175102466887 0loss/SRqqt3/detection_loss‚1.763651202100196 .loss/FBt3/detection_loss‚2.542326807975769 0loss/FRt4/assignment_loss‚0.9781174090775576 2loss/SRqqt3/assignment_loss‚1.0567793124687992 1loss/FBt4/assignment_loss‚0.19023681601968512 .loss/FRt1/detection_loss‚2.104206323027611 0loss/SRqqt2/detection_loss‚2.186151738229551 %loss/total_loss‚41.72675358148505 _step‚839 /loss/FRt3/detection_loss‚2.3474152834843034 0loss/FRt1/assignment_loss‚3.2382408690452578 -loss/FRt3/assignment_loss‚2.6614803114663 .loss/FBt1/detection_loss‚4.074452412128449 /loss/FBt4/detection_loss‚0.4271932444084556 epoch‚8 /loss/FRt2/assignment_loss‚2.740576594164877 1loss/SRqqt4/detection_loss‚1.0072423952130172 .loss/FBt2/detection_loss‚2.947631537914276 2loss/SRqqt1/assignment_loss‚1.6531725597050455 1loss/SRqqt2/assignment_loss‚1.992724980570768 $ _timestamp‚1.788424649088707e+09)ÓàV[Š&Õ 1loss/SRqqt2/assignment_loss‚1.992724980570768 $ _timestamp‚1.788424649088707e+09 2loss/SRqqt4/assignment_loss‚0.8740000726726765 0loss/FBt1/assignment_loss‚0.1297996485605836 trainer/global_step‚10199 1loss/FBt2/assignment_loss‚0.34888488723441924 1loss/FBt3/assignment_loss‚0.12270193605218083 0loss/SRqqt1/detection_loss‚1.701277063290278 _runtime‚8148.826164366 /loss/FRt2/detection_loss‚3.0185726664283057 /loss/FRt4/detection_loss‚1.6199175102466887 0loss/SRqqt3/detection_loss‚1.763651202100196 .loss/FBt3/detection_loss‚2.542326807975769 0loss/FRt4/assignment_loss‚0.9781174090775576 2loss/SRqqt3/assignment_loss‚1.0567793124687992 1loss/FBt4/assignment_loss‚0.19023681601968512 .loss/FRt1/detection_loss‚2.104206323027611 0loss/SRqqt2/detection_loss‚2.186151738229551 %loss/total_loss‚41.72675358148505 _step‚839 /loss/FRt3/detection_loss‚2.3474152834843034 0loss/FRt1/assignment_loss‚3.2382408690452578 -loss/FRt3/assignment_loss‚2.6614803114663 .loss/FBt1/detection_loss‚4.074452412128449 /loss/FBt4/detection_loss‚0.4271932444084556 epoch‚8 /loss/FRt2/assignment_loss‚2.740576594164877 1loss/SRqqt4/detection_loss‚1.0072423952130172 .loss/FBt2/detection_loss‚2.947631537914276 2loss/SRqqt1/assignment_loss‚1.6531725597050455Çò,šj‹&‚(:’ ÔãäÔ©ž~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bŠ¸uŒ&‚(:ê ÔãäÔÙί€( disk.nvme0c0n1.out‚34900.96337890625( proc.memory.availableMB‚ 951.94921875* proc.memory.percent‚16.921377182006836 disk.dm-0.in‚4.6875# disk.dm-0.out‚34900.96337890625" proc.memory.rssMB‚ 2772.3984375* disk./.usagePercent‚17.486322490274173 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ memory_percent‚94.18976306915283 cpu‚25.345831069807083 disk.drbd2265.out‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 network.sent‚ 2.466343e+07 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 439.01171875 disk.rbd1.out‚ 4.29296875 disk.sda.out‚ 439.01171875 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 439.01171875 proc.cpu.threads‚17 network.recv‚ 2.980539e+06% disk./.usageGB‚260.55884170532227& disk.nvme0n1.out‚34900.96337890625Ä9ÍG&‚(:’ ããäÔ°Ÿ~ gpu.0.gpu‚45.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.973200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X5%ü^Ž&‚(:Ó ããäÔåÚ“€ disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚17.486849097772104 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.dm-1.out‚ 439.45703125 disk.nvme0c0n1.in‚4.6875% disk./.usageGB‚260.56668853759766 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.47713e+07 disk.sda3.out‚ 439.45703125 disk.drbd2265.out‚0 disk.rbd1.out‚ 4.29296875$ proc.memory.availableMB‚951.9375* proc.memory.percent‚16.921377182006836 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.6875" disk.nvme0c0n1.out‚ 34957.15625$ memory_percent‚94.18983459472656 proc.cpu.threads‚17 disk.sda.in‚ 126.77734375 disk.sda.out‚ 439.45703125 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚ 2.985858e+06 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.out‚ 34957.15625 cpu‚25.345321935205117 disk.sda2.out‚0 disk.dm-0.out‚ 34957.15625 disk.drbd2266.in‚0íåÊ2&- _wandbruntime‚8185  _runtime‚8185}+õÊ&Ä ' lr-AdamW/pg2‚1.1636996587443654e-05 trainer/global_step‚10249 % _timestamp‚1.7884246854074159e+09 _runtime‚8185.145036946 _step‚840 ' lr-AdamW/pg1‚1.1636996587443654e-05e]¸Ï‘&É _step‚840 ' lr-AdamW/pg1‚1.1636996587443654e-05 ' lr-AdamW/pg2‚1.1636996587443654e-05 trainer/global_step‚10249 % _timestamp‚1.7884246854074159e+09 _runtime‚8185.145036946ÈÍî,2’&- _wandbruntime‚8185  _runtime‚8185Žáh]“&âO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49019897 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚10249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9006606840320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 % _timestamp‚1.7884246854081678e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 [IDeviceStatsMonitor.on_train_batch_st² w‹art/active_bytes.small_pool.allocated‚ 8586655400960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8586655400960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589125464898012 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596197895193088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87005718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8185.145878845 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49019456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036860711936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193530778 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136025172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487644 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604784550594048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006774658560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608HÕ†í'”&çO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8586655400960 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8586655400960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589125464898012 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚580571076410368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚596197895193088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚580571093449728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87005718 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8185.145878845 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49019456 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚660036860711936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193530778 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136025172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8555042925400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8554388487644 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚604784550594048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87005716 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚443 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9006774658560 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚604783876228608 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193550816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149117822 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149117822 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49019897 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 trainer/global_step‚10249 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9006606840320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 % _timestamp‚1.7884246854081678e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚596197877190144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430Éé „62•&- _wandbruntime‚8185  _runtime‚8185ú³Û1ò&–&ìM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660099369120768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193549232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555204922608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723074048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 % _timestamp‚1.7884246860941815e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701134 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818466304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604841723074048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87013884 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚842 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136038308 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660099181570048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193549232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136037859 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149131836 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838°cð€A—&ñM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 342681068 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651091736791040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚580625980540928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8555204922608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342681068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149131836 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚604841723074048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 % _timestamp‚1.7884246860941815e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342701134 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8586818466304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚604841723074048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.`²U?¶freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87013884 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8555859360572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 _step‚842 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193569296 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚596254248231936 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136038308 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚660099181570048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651091712850944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚604841048695296 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193549232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136037859 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚596254248231936 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149131836 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342701134 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149131838 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8587474842112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚660099369120768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193549232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589181185463536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÊkøÈH2˜&- _wandbruntime‚8185  _runtime‚8185…(ž`™&Ú &loss/total_loss‚42.099207744819154 1loss/FBt1/assignment_loss‚0.06379764606910092 1loss/FBt2/assignment_loss‚0.29458108736963334 0loss/FBt3/assignment_loss‚0.7281411874719197 1loss/SRqqt1/detection_loss‚1.9316503057877223 trainer/global_step‚10249 2loss/SRqqt4/assignment_loss‚0.9001363227143884 /loss/FRt2/detection_loss‚3.8028482654515434 epoch‚8 0loss/FRt3/assignment_loss‚2.1456127365430198 0loss/SRqqt4/detection_loss‚1.131206264586321 2loss/SRqqt1/assignment_loss‚1.1864519271822203 2loss/SRqqt3/assignment_loss‚1.1687296860665084 /loss/FRt1/detection_loss‚2.5213682745632373 1loss/SRqqt3/detection_loss‚1.8474448987841605 /loss/FBt2/detection_loss‚2.5916879962791097 _runtime‚8185.833232946 0loss/FRt1/assignment_loss‚3.2290597291369187 0loss/FRt4/assignment_loss‚1.7844796094027433 1loss/SRqqt2/detection_loss‚2.3234765000881685 _step‚843 .loss/FRt3/detection_loss‚2.152287536976384 .loss/FBt1/detection_loss‚2.844465438808714 % _timestamp‚1.7884246860958185e+09 1loss/FBt4/assignment_loss‚0.17582691898525135 0loss/FRt2/assignment_loss‚2.6580764055252075 2loss/SRqqt2/assignment_loss‚2.2223132560330052 .loss/FRt4/detection_loss‚1.665718159739944 .loss/FBt3/detection_loss‚2.282226584851742 0loss/FBt4/detection_loss‚0.447621006402187055fCveš&ß 0loss/SRqqt4/detection_loss‚1.131206264586321 2loss/SRqqt1/assignment_loss‚1.1864519271822203 2loss/SRqqt3/assignment_loss‚1.1687296860665084 /loss/FRt1/detection_loss‚2.5213682745632373 1loss/SRqqt3/detection_loss‚1.8474448987841605 /loss/FBt2/detection_loss‚2.5916879962791097 _runtime‚8185.833232946 0loss/FRt1/assignment_loss‚3.2290597291369187 0loss/FRt4/assignment_loss‚1.7844796094027433 1loss/SRqqt2/detection_loss‚2.3234765000881685 _step‚843 .loss/FRt3/detection_loss‚2.152287536976384 .loss/FBt1/detection_loss‚2.844465438808714 % _timestamp‚1.7884246860958185e+09 1loss/FBt4/assignment_loss‚0.17582691898525135 0loss/FRt2/assignment_loss‚2.6580764055252075 2loss/SRqqt2/assignment_loss‚2.2223132560330052 .loss/FRt4/detection_loss‚1.665718159739944 .loss/FBt3/detection_loss‚2.282226584851742 0loss/FBt4/detection_loss‚0.44762100640218705 &loss/total_loss‚42.099207744819154 1loss/FBt1/assignment_loss‚0.06379764606910092 1loss/FBt2/assignment_loss‚0.29458108736963334 0loss/FBt3/assignment_loss‚0.7281411874719197 1loss/SRqqt1/detection_loss‚1.9316503057877223 trainer/global_step‚10249 2loss/SRqqt4/assignment_loss‚0.9001363227143884 /loss/FRt2/detection_loss‚3.8028482654515434 epoch‚8 0loss/FRt3/assignment_loss‚2.1456127365430198Ëæp¼ú’›&‚(:‡ òãäÔ‰Ž¿~ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K\hœ&‚(:Ý òãäÔ“›È€ disk.sda1.out‚0" proc.memory.rssMB‚ 2772.3984375 cpu‚25.344749651612904 disk.dm-0.out‚35001.54296875 disk.sda1.in‚0# disk.nvme0n1.out‚35001.54296875 disk.nvme0c0n1.in‚4.6875 network.sent‚ 2.4777276e+07 disk.drbd2080.in‚0 disk.dm-1.out‚ 440.0703125% disk.nvme0c0n1.out‚35001.54296875 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda3.out‚ 440.0703125' proc.memory.availableMB‚ 951.0390625 proc.cpu.threads‚17* disk./.usagePercent‚17.487414362407705% disk./.usageGB‚260.57511138916016 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375* proc.memory.percent‚16.921377182006836 network.recv‚ 2.990283e+06 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0# memory_percent‚94.1953182220459 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375 disk.sda.out‚ 440.0703125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0¥SK’&‚(:‡ ääÔ׶™~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚131.369' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú –ùpž&‚(:å ääÔÖŒ™% disk./.usageGB‚260.58938217163086 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 440.5703125$ memory_percent‚94.19231414794922 proc.cpu.threads‚17 network.sent‚ 2.4884907e+07 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚35049.78759765625 disk.sda3.out‚ 440.5703125 disk.drbd2080.in‚0* disk./.usagePercent‚17.488372086593298 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 cpu‚25.344208555470615 disk.nvme0c0n1.in‚4.6875# disk.dm-0.out‚35049.78759765625 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.921377182006836 network.recv‚ 2.995586e+06 disk.drbd2266.in‚0 disk.dm-1.out‚ 440.5703125 disk.rbd0.out‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2772.3984375 disk.drbd1716.out‚0& disk.nvme0n1.out‚35049.78759765625 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 4.29296875% proc.memory.availableMB‚ 951.53125 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 Ïù’Ÿ&‚(:‡ ääÔñ¤¹~ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5834 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬!òq &‚(:æ ääÔ¹­¿ disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.sda.out‚ 441.23828125 disk.sda1.in‚0( disk.nvme0c0n1.out‚35089.67822265625 disk.drbd2266.in‚0 disk.drbd1716.in‚0# disk.dm-0.out‚35089.67822265625$ memory_percent‚94.19231414794922 network.sent‚ 2.4891412e+07 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.sda2.in‚0 network.recv‚ 3.000469e+06$ disk./.usageGB‚260.6277160644531 disk.rbd0.in‚0 disk.sda3.out‚ 441.23828125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.49094470630488 disk.dm-1.out‚ 441.23828125& disk.nvme0n1.out‚35089.67822265625 disk.dm-0.in‚4.6875% proc.memory.availableMB‚ 951.53125* proc.memory.percent‚16.921377182006836 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.6875 cpu‚25.343701743397407 disk.rbd0.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2772.3984375x¡]2¡&- _wandbruntime‚8221  _runtime‚8221í?4aÊ¢&Ä ' lr-AdamW/pg1‚1.1027445076614576e-05 ' lr-AdamW/pg2‚1.1027445076614576e-05 trainer/global_step‚10299 % _timestamp‚1.7884247221512706e+09 _runtime‚8221.888777026 _step‚844¯ý…žÏ£&É _step‚844 ' lr-AdamW/pg1‚1.1027445076614576e-05 ' lr-AdamW/pg2‚1.1027445076614576e-05 trainer/global_step‚10299 % _timestamp‚1.7884247221512706e+09 _runtime‚8221.888777026Ìä ‚2¤&- _wandbruntime‚8221  _runtime‚8221“üŠ#è'¥&âO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599015529132544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194456830 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49246799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599015547135488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136660917 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513976832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984844288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9049692467712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚8221.889633031 _step‚845 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149818620 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599015529132544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49246352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591912178242536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344275450 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87414118 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚583316282938368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188342272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884247221519365e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430ñÚ!Šÿ"¦&çO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136660917 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚607642513976832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8626984844288 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9049692467712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 _runtime‚8221.889633031 _step‚845 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149818620 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚599015529132544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49246352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344295490 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚591912178242536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344275450 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87414118 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚583316282938368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚607643188342272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 % _timestamp‚1.7884247221519365e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87414116 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599015529132544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194456830 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8626984844288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚607642513976832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49246799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599015547135488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚447 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521Íàö3þ2§&- _wandbruntime‚8222  _runtime‚8222q*b×ò&¨&ìM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149832638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9050387997696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚846 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149832636 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 194495348 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49251246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599071882171392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599071900174336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136673090 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050549511168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686443520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596694699980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663224823185408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804272128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8596040262016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654174273674240 _runtime‚8222.581266981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49250806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591967227330944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194495348 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149832638 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607700360822272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460647936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚583371204108288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884247228432264e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599071882171392 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚599071900174336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136673090 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9050549511168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚607699686443520 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚440 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚654174249734144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8596694699980 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591967898808268 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚663224823185408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8627804272128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149832636 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8596040262016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚10299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194475284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344327986 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚654174273674240 _runtime‚8222.581266981 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚607700360822272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87422286 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87422284 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49250806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591967227330944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8627804272128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194495348 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149832638 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚607700360822272 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8628460647936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚583371204108288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 344307920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344307920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194475284 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 % _timestamp‚1.7884247228432264e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Îb€2ª&- _wandbruntime‚8222  _runtime‚8222ß û$Z«&Ô .loss/FRt4/detection_loss‚1.450333372610858 .loss/FBt2/detection_loss‚3.832762375473976 2loss/SRqqt1/assignment_loss‚1.3598984004381824 1loss/SRqqt3/detection_loss‚1.5931187633138437 &loss/total_loss‚43.241382282965226 _runtime‚8222.582558547 1loss/FBt3/assignment_loss‚0.42500118513089546 2loss/SRqqt2/assignment_loss‚1.6813047660454628 /loss/FBt2/assignment_loss‚0.634513481054455 .loss/FRt1/detection_loss‚2.346158333915345 0loss/FRt2/assignment_loss‚3.8096252509525845 0loss/FRt4/assignment_loss‚1.2905861432269468 1loss/FBt1/assignment_loss‚0.04309834085870534 /loss/FBt3/detection_loss‚2.0818390771746635 0loss/FRt1/assignment_loss‚2.7446294097190207 /loss/FRt3/assignment_loss‚3.005569625468481 2loss/SRqqt4/assignment_loss‚0.9244753233120813 0loss/FBt4/assignment_loss‚0.1517831558397201 .loss/FBt1/detection_loss‚2.425021640956402 _step‚847 /loss/FBt4/detection_loss‚0.8336957822062282 epoch‚8 trainer/global_step‚10299 .loss/FRt2/detection_loss‚3.050206940088953 1loss/SRqqt1/detection_loss‚1.8043570562320597 0loss/SRqqt2/detection_loss‚2.750138698085662 1loss/SRqqt4/detection_loss‚0.9791121422857433 % _timestamp‚1.7884247228451328e+09 2loss/SRqqt3/assignment_loss‚1.5375094125763729 .loss/FRt3/detection_loss‚2.486643605998584Ž‹ñ¸_¬&Ù 2loss/SRqqt1/assignment_loss‚1.3598984004381824 1loss/SRqqt3/detection_loss‚1.5931187633138437 &loss/total_loss‚43.241382282965226 _runtime‚8222.582558547 1loss/FBt3/assignment_loss‚0.42500118513089546 2loss/SRqqt2/assignment_loss‚1.6813047660454628 /loss/FBt2/assignment_loss‚0.634513481054455 .loss/FRt1/detection_loss‚2.346158333915345 0loss/FRt2/assignment_loss‚3.8096252509525845 0loss/FRt4/assignment_loss‚1.2905861432269468 1loss/FBt1/assignment_loss‚0.04309834085870534 /loss/FBt3/detection_loss‚2.0818390771746635 0loss/FRt1/assignment_loss‚2.7446294097190207 /loss/FRt3/assignment_loss‚3.005569625468481 2loss/SRqqt4/assignment_loss‚0.9244753233120813 0loss/FBt4/assignment_loss‚0.1517831558397201 .loss/FBt1/detection_loss‚2.425021640956402 _step‚847 /loss/FBt4/detection_loss‚0.8336957822062282 epoch‚8 trainer/global_step‚10299 .loss/FRt2/detection_loss‚3.050206940088953 1loss/SRqqt1/detection_loss‚1.8043570562320597 0loss/SRqqt2/detection_loss‚2.750138698085662 1loss/SRqqt4/detection_loss‚0.9791121422857433 % _timestamp‚1.7884247228451328e+09 2loss/SRqqt3/assignment_loss‚1.5375094125763729 .loss/FRt3/detection_loss‚2.486643605998584 .loss/FRt4/detection_loss‚1.450333372610858 .loss/FBt2/detection_loss‚3.832762375473976ÏW«ý,­&‚(:’ ŸääÔÊΆ~ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.765' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.352999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d¿¦‰b®&‚(:× ŸääÔÑÜ disk.rbd1.out‚ 4.29296875 disk.sda1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚35138.7177734375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.in‚0" disk.dm-0.out‚35138.7177734375$ memory_percent‚94.19097900390625 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 441.609375" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 cpu‚25.34314052717597 network.sent‚ 2.489672e+07 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚4.6875 network.recv‚ 3.00403e+06) disk./.usagePercent‚17.49126599576035 disk.dm-0.in‚4.6875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚35138.7177734375 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚260.6325035095215 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 441.609375 disk.sda.out‚ 441.609375 disk.sda2.in‚0" proc.memory.availableMB‚951.75>wÚä’¯&‚(:‡ ®ääÔµ¢¹~ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.978' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.5956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹+Ýci°&‚(:Þ ®ääÔÞÆ€ disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0* proc.memory.percent‚16.921377182006836 disk.drbd1716.out‚0% disk.nvme0n1.out‚35178.1357421875 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.29296875 cpu‚25.34264017366675 disk.drbd2080.out‚0 network.recv‚ 3.010449e+06* disk./.usagePercent‚17.491399119678157 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 442.26171875 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 442.26171875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 442.26171875! proc.memory.availableMB‚951.5" proc.memory.rssMB‚ 2772.3984375 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875" disk.dm-0.out‚35178.1357421875 proc.cpu.threads‚17' disk.nvme0c0n1.out‚35178.1357421875 disk.sda2.out‚0 disk.drbd2265.in‚0# memory_percent‚94.1925048828125 network.sent‚ 2.5005421e+07$ disk./.usageGB‚260.6344871520996 disk.sda1.in‚0 disk.drbd2080.in‚0ÊØb2±&- _wandbruntime‚8258  _runtime‚8258 —à†É²&à _runtime‚8258.631752796 _step‚848 ' lr-AdamW/pg1‚1.0433172480769694e-05 ' lr-AdamW/pg2‚1.0433172480769694e-05 trainer/global_step‚10349 $ _timestamp‚1.788424758894255e+09Ég½Î³&È ' lr-AdamW/pg1‚1.0433172480769694e-05 ' lr-AdamW/pg2‚1.0433172480769694e-05 trainer/global_step‚10349 $ _timestamp‚1.788424758894255e+09 _runtime‚8258.631752796 _step‚848ÐVs½2´&- _wandbruntime‚8258  _runtime‚8258ÚÒ§µ&âO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151725056 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092635540992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8667970650112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666287611991040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657195159819776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092476111360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137294442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666287795360768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610501826090496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195382882 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚601833199077888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49471926 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VBDeviceStatsMonitor.on_train_íãI«Abatch_start/active_bytes.all.allocated‚610501826090496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601833181074944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137294882 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87822516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 % _timestamp‚1.7884247588949244e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195402920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586061506505728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚601833181074944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636713604216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚594698220109944 _runtime‚8258.632641673 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627012608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150519420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088;ò“}í'¶&çO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195402920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49472364 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137294882 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8667970650112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150519422 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345922342 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8668627012608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87822516 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚594697548632828 % _timestamp‚1.7884247588949244e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195402920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚586061506505728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚601833181074944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚586061489466368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8636713604216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚849 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150519420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87822518 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚594698220109944 _runtime‚8258.632641673 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8668627012608 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150519420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚601833199077888 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610501151725056 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150519422 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10349 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9092635540992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8667970650112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚666287611991040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8636059166460 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚657195159819776 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9092476111360 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610501151725056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137294442 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666287795360768 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610501826090496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚657195135879680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195382882 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚601833199077888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49471926 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610501826090496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601833181074944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8668790077952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093497822208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039388 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666350308379648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669446453760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093334211584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195421400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚850 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928 % _timestamp‚1.7884247595841072e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668790077952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666350120828928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137307122ÉlO‹¹&ñM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601889534113792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345954838 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87830686 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9093334211584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150533438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601889552116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195421400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195421400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601889552116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195401336 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚850 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8669446453760 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195401336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚586116393596928 % _timestamp‚1.7884247595841072e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345954838 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345934772 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8668790077952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚586116410636288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345934772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚666350120828928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137307122 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49476438 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150533436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49476885 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150533438 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8668790077952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20ˆ|™l 064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9093497822208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8637530039388 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚610558998570496 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666350308379648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601889534113792 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150533436 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8669446453760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414ÒpŒÆ›2º&- _wandbruntime‚8259  _runtime‚8259”é[…Y»&Ó 1loss/FBt2/assignment_loss‚0.20617096811191812 1loss/FBt4/assignment_loss‚0.10854703266314787 .loss/FRt1/detection_loss‚2.248494479466568 0loss/SRqqt2/detection_loss‚2.176797227263451 $ _timestamp‚1.788424759586292e+09 _runtime‚8259.323746607 2loss/FBt3/assignment_loss‚0.012122741589943567 0loss/SRqqt3/detection_loss‚1.779969372972846 1loss/SRqqt3/assignment_loss‚1.221628233604133 /loss/FBt4/detection_loss‚0.5505970679620408 _step‚851 2loss/SRqqt1/assignment_loss‚1.4106840380423127 -loss/FRt4/detection_loss‚1.56568791010246 %loss/total_loss‚41.17911448666729 2loss/SRqqt2/assignment_loss‚1.6138786432147025 /loss/FRt3/detection_loss‚2.3635461495622345 /loss/FBt1/detection_loss‚2.4293203006188073 trainer/global_step‚10349 .loss/FRt2/assignment_loss‚3.21604414837701 0loss/FRt3/assignment_loss‚2.4476986022099205 0loss/FRt4/assignment_loss‚1.7586363602657706 2loss/SRqqt4/assignment_loss‚0.9166799333519661 /loss/FRt2/detection_loss‚2.6502041135515486 1loss/SRqqt1/detection_loss‚2.0380303256568455 1loss/SRqqt4/detection_loss‚1.1167108993404187 .loss/FBt2/detection_loss‚2.483482321103414 /loss/FBt3/detection_loss‚2.9888209899266562 epoch‚8 /loss/FRt1/assignment_loss‚3.473298580809073 0loss/FBt1/assignment_loss‚0.4020640469000985î©!é^¼&Ø -loss/FRt4/detection_loss‚1.56568791010246 %loss/total_loss‚41.17911448666729 2loss/SRqqt2/assignment_loss‚1.6138786432147025 /loss/FRt3/detection_loss‚2.3635461495622345 /loss/FBt1/detection_loss‚2.4293203006188073 trainer/global_step‚10349 .loss/FRt2/assignment_loss‚3.21604414837701 0loss/FRt3/assignment_loss‚2.4476986022099205 0loss/FRt4/assignment_loss‚1.7586363602657706 2loss/SRqqt4/assignment_loss‚0.9166799333519661 /loss/FRt2/detection_loss‚2.6502041135515486 1loss/SRqqt1/detection_loss‚2.0380303256568455 1loss/SRqqt4/detection_loss‚1.1167108993404187 .loss/FBt2/detection_loss‚2.483482321103414 /loss/FBt3/detection_loss‚2.9888209899266562 epoch‚8 /loss/FRt1/assignment_loss‚3.473298580809073 0loss/FBt1/assignment_loss‚0.4020640469000985 1loss/FBt2/assignment_loss‚0.20617096811191812 1loss/FBt4/assignment_loss‚0.10854703266314787 .loss/FRt1/detection_loss‚2.248494479466568 0loss/SRqqt2/detection_loss‚2.176797227263451 $ _timestamp‚1.788424759586292e+09 _runtime‚8259.323746607 2loss/FBt3/assignment_loss‚0.012122741589943567 0loss/SRqqt3/detection_loss‚1.779969372972846 1loss/SRqqt3/assignment_loss‚1.221628233604133 /loss/FBt4/detection_loss‚0.5505970679620408 _step‚851 2loss/SRqqt1/assignment_loss‚1.4106840380423127ÓQÏ{½&‚(:… ½ääÔÝ™~ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘RöPj¾&‚(:ß ½ääÔÉæ•€ disk.sda.in‚ 126.77734375 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 442.8984375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0$ memory_percent‚94.19293403625488' proc.memory.availableMB‚ 951.4296875 network.recv‚ 3.014483e+06 disk.dm-0.in‚4.6875" disk.dm-0.out‚35217.2978515625 disk.sda.out‚ 442.8984375 disk.sda1.in‚0% disk.nvme0n1.out‚35217.2978515625 disk.nvme0c0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375( disk./.usagePercent‚17.4923153706432 disk.drbd2080.in‚0 disk.dm-1.out‚ 442.8984375 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd2265.out‚0 network.sent‚ 2.5011285e+07 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚35217.2978515625* proc.memory.percent‚16.921377182006836 cpu‚25.3420859303218$ disk./.usageGB‚260.6481399536133 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0“óhœ¿&‚(:‘ ÌääÔ®ä•~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.989999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'¯#mmÀ&‚(:â ÌääÔ¾§„€$ proc.memory.availableMB‚951.4375 cpu‚25.34155931334255 disk.rbd0.out‚0# disk.dm-0.out‚35265.75341796875 disk.drbd2266.out‚0 disk.sda1.in‚0$ disk./.usageGB‚260.6537437438965 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 443.53515625 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.6875 disk.rbd1.out‚ 4.29296875 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0$ memory_percent‚94.19288635253906 disk.rbd0.in‚0 disk.drbd2266.in‚0 network.recv‚ 3.020912e+06 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 443.53515625' disk./.usagePercent‚17.492691445711 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 443.53515625 disk.sda1.out‚0& disk.nvme0n1.out‚35265.75341796875* proc.memory.percent‚16.921377182006836 network.sent‚ 2.5120179e+07( disk.nvme0c0n1.out‚35265.75341796875 disk.drbd2080.out‚0^¬!Á&‚(:’ ÛääÔð¸ò} gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.985' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.797000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂBtsÂ&‚(:è ÛääÔæÝ€ disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.6875& disk.nvme0n1.out‚35313.63037109375# disk.dm-0.out‚35313.63037109375 disk.dm-1.out‚ 443.94921875* disk./.usagePercent‚17.493239301834276 disk.nvme0c0n1.in‚4.6875 proc.cpu.threads‚17 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚35313.63037109375 disk.dm-0.in‚4.6875 disk.sda1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 951.4296875 cpu‚25.340999333919548 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 network.sent‚ 2.5125491e+07 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚94.19293403625488 disk.drbd2266.in‚0 disk.sda.out‚ 443.94921875 disk.sda2.out‚0 disk.sda3.out‚ 443.94921875 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836$ disk./.usageGB‚260.6619071960449 disk.drbd2265.out‚0 network.recv‚ 3.02449e+06 disk.sda3.in‚ 126.77734375䚉2Ã&- _wandbruntime‚8295  _runtime‚8295X¦ÖÇÄ&Á & lr-AdamW/pg2‚9.854302645427217e-06 trainer/global_step‚10399 $ _timestamp‚1.788424795605574e+09 _runtime‚8295.343117262 _step‚852 & lr-AdamW/pg1‚9.854302645427217e-06Æ¡dÌÅ&Æ $ _timestamp‚1.788424795605574e+09 _runtime‚8295.343117262 _step‚852 & lr-AdamW/pg1‚9.854302645427217e-06 & lr-AdamW/pg2‚9.854302645427217e-06 trainer/global_step‚10399Ô}1V2Æ&- _wandbruntime‚8295  _runtime‚8295=€¿Õè'Ç&âO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884247956063273e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505868 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669413408109056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196308934 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669413214253568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88230916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9135711406080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8708956455936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8709612818432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚660277672762880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137928936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196328972 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588806695994368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463838720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49698471 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789473280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 _step‚853 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612818432 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137929389 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49698020 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 8618535218Ήí'È&çO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137929389 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88230918 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151220220 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49698020 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 % _timestamp‚1.7884247956063273e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8676894505868 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151220220 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8708956455936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669413408109056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196308934 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚669413214253568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88230916 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660277696702976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613360463838720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151220222 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9135711406080 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚597483590500236 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588806713033728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8708956455936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347529154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚604650833017344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚604650851020288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8709612818432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚660277672762880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137928936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196328972 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613359789473280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588806695994368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚613360463838720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347549194 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49698471 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613359789473280 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚604650851020288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚604650833017344 _step‚853 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8709612818432Õ„ÍÛ2É&- _wandbruntime‚8296  _runtime‚8296˜8£%Ê&ìM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636318720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775883776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚660339323500544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775883776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8677710940832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8710432259584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/ac£z¾±Ö tive_bytes.large_pool.allocated‚604707204059136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196347452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432259584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136389497344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604707186056192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588861600124928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347561624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88239086 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378796 trainer/global_step‚10399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327388 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604707186056192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49702505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636318720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669475908937216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136561496576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137941589 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _step‚854eÇ÷&Ë&ñM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9136389497344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚604707186056192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151234238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588861600124928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347561624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196327388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88239086 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8678365378796 trainer/global_step‚10399 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347581690 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151234236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196327388 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚604707186056192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49702505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚613417636318720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347561624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669475908937216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9136561496576 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚604707204059136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347581690 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137941589 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151234236 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _step‚854 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚613417636318720 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8709775883776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚660339323500544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8709775883776 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196347452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚449 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8677710940832 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8710432259584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚604707204059136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196347452 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8710432259584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151234238 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚597539311065760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÖT·2Ì&- _wandbruntime‚8296  _runtime‚8296ö_«]Í&× /loss/FBt4/detection_loss‚0.5668280293380744 epoch‚8 _runtime‚8296.044807106 0loss/FRt3/assignment_loss‚2.6708156186707166 0loss/FBt1/assignment_loss‚0.2520729502295415 1loss/SRqqt2/detection_loss‚3.0413521158695223 /loss/FBt1/detection_loss‚3.0846690102057024 0loss/FRt2/assignment_loss‚3.2925084078753435 0loss/FRt4/assignment_loss‚1.4123988164232133 /loss/FRt3/detection_loss‚2.2922830405283947 1loss/SRqqt1/detection_loss‚2.1437145940959454 0loss/SRqqt4/detection_loss‚1.012618059190837 % _timestamp‚1.7884247963073673e+09 .loss/FRt1/detection_loss‚2.318732219211983 /loss/FRt4/detection_loss‚1.5626204601944762 trainer/global_step‚10399 2loss/SRqqt1/assignment_loss‚1.4955254666507245 1loss/SRqqt3/assignment_loss‚1.247506988590414 0loss/FBt4/assignment_loss‚0.1480919363840722 /loss/FBt3/detection_loss‚2.7629988193511963 $loss/total_loss‚43.3547009123097 2loss/SRqqt2/assignment_loss‚1.6168725446425378 0loss/FBt2/assignment_loss‚0.4892302841763005 1loss/FBt3/assignment_loss‚0.08890210551180644 _step‚855 0loss/FRt1/assignment_loss‚3.6686210307207974 2loss/SRqqt4/assignment_loss‚0.8237796681001782 /loss/FRt2/detection_loss‚2.9476524149930037 1loss/SRqqt3/detection_loss‚1.6420328164642508 /loss/FBt2/detection_loss‚2.7728735148906707Ÿž´bÎ&Ü 0loss/FBt4/assignment_loss‚0.1480919363840722 /loss/FBt3/detection_loss‚2.7629988193511963 $loss/total_loss‚43.3547009123097 2loss/SRqqt2/assignment_loss‚1.6168725446425378 0loss/FBt2/assignment_loss‚0.4892302841763005 1loss/FBt3/assignment_loss‚0.08890210551180644 _step‚855 0loss/FRt1/assignment_loss‚3.6686210307207974 2loss/SRqqt4/assignment_loss‚0.8237796681001782 /loss/FRt2/detection_loss‚2.9476524149930037 1loss/SRqqt3/detection_loss‚1.6420328164642508 /loss/FBt2/detection_loss‚2.7728735148906707 /loss/FBt4/detection_loss‚0.5668280293380744 epoch‚8 _runtime‚8296.044807106 0loss/FRt3/assignment_loss‚2.6708156186707166 0loss/FBt1/assignment_loss‚0.2520729502295415 1loss/SRqqt2/detection_loss‚3.0413521158695223 /loss/FBt1/detection_loss‚3.0846690102057024 0loss/FRt2/assignment_loss‚3.2925084078753435 0loss/FRt4/assignment_loss‚1.4123988164232133 /loss/FRt3/detection_loss‚2.2922830405283947 1loss/SRqqt1/detection_loss‚2.1437145940959454 0loss/SRqqt4/detection_loss‚1.012618059190837 % _timestamp‚1.7884247963073673e+09 .loss/FRt1/detection_loss‚2.318732219211983 /loss/FRt4/detection_loss‚1.5626204601944762 trainer/global_step‚10399 2loss/SRqqt1/assignment_loss‚1.4955254666507245 1loss/SRqqt3/assignment_loss‚1.247506988590414×péŽÈÏ&‚(:… êääÔŒ‰Ö gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚137.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ]BiÐ&‚(:Þ êääÔøèè disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875$ memory_percent‚94.19312477111816 proc.cpu.threads‚17 disk.sda.out‚ 444.59375 disk.drbd2265.out‚0 network.sent‚ 2.513152e+07 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 444.59375" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 cpu‚25.34049878859377& disk.nvme0n1.out‚35356.32958984375 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0$ disk./.usageGB‚260.6631393432617 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0# disk.dm-0.out‚35356.32958984375 network.recv‚ 3.028868e+06 disk.sda1.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35356.32958984375 disk.sda2.out‚0 disk.sda3.out‚ 444.59375 disk.dm-1.in‚ 126.77734375* disk./.usagePercent‚17.493321992267834 disk.nvme0n1.in‚4.6875 disk.dm-0.in‚4.6875' proc.memory.availableMB‚ 951.3984375 disk.rbd1.out‚ 4.29296875 disk.sda2.in‚0 disk.drbd2266.in‚0|)ð(‘Ñ&‚(:† ùääÔÍõˆ~ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚178.655' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.731 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(¬öˆgÒ&‚(:Ü ùääÔÙ‘Ï$ disk./.usageGB‚260.6875419616699 disk.drbd2080.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚445.0625 disk.sda2.out‚0$ memory_percent‚94.19312477111816 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 cpu‚25.339978288737363& disk.nvme0n1.out‚35398.11083984375 disk.sda3.out‚445.0625 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚35398.11083984375 disk.dm-1.out‚445.0625( disk.nvme0c0n1.out‚35398.11083984375' proc.memory.availableMB‚ 951.3984375) disk./.usagePercent‚17.49495967246437 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 network.sent‚ 2.5240077e+07 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875 network.recv‚ 3.034673e+06 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17¶å6C2Ó&- _wandbruntime‚8332  _runtime‚8332‰ú7¿ÇÔ&Á & lr-AdamW/pg2‚9.290956206193225e-06 trainer/global_step‚10449 % _timestamp‚1.7884248323017771e+09 _runtime‚ 8332.03929452 _step‚856 & lr-AdamW/pg1‚9.290956206193225e-06%5z,ÌÕ&Æ _runtime‚ 8332.03929452 _step‚856 & lr-AdamW/pg1‚9.290956206193225e-06 & lr-AdamW/pg2‚9.290956206193225e-06 trainer/global_step‚10449 % _timestamp‚1.7884248323017771e+09Ø*|œ2Ö&- _wandbruntime‚8332  _runtime‚8332°ðˆø×&áO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9178581913600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 8332.04017426 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101586944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663360233586176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49925502 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138564818 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9178747634688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600270303844760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942261760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672538791559680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197255024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427221504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138565274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349156006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151921022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598624256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMo ã–ïnitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚857 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234986 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚607468484959744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49925956IŒSì'Ø&æO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚591551902522368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8749942261760 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88639316 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚672538791559680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚616218427221504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197255024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚10449 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚616218427221504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138565274 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349156006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8749942261760 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151921022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88639318 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8750598624256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591551919561728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197234986 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚857 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 197234986 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚607468484959744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49925956 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8750598624256 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151921020 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9178581913600 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _runtime‚ 8332.04017426 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚616219101586944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663360233586176 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600269632367644 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151921022 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49925502 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151921020 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚616219101586944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663360209646080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚607468502962688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 349176046 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138564818 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9178747634688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600270303844760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680ÙÏÐ'2Ù&- _wandbruntime‚8332  _runtime‚8332ëÛJRò&Ú&ìM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601334032896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591606823692288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚616276274066944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410492 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179618385408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601502709248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8750761689600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49930478 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718204 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179473649152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚858 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138577529 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151935038 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197253440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964ay†ñ÷&Û&ñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚591606806652928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138577529 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151935038 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 197253440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8751418065408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚672601334032896 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚591606823692288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚616276274066944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600326024410492 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8718546280240 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚607524856001536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9179618385408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197273504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚672601502709248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197253440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 349208542 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8750761689600 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49930478 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 349208542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8719200718204 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9179473649152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 _step‚858ÚSI»ø2Ü&- _wandbruntime‚8332  _runtime‚8332ñÉ~¬§Ý&Ô /loss/FBt2/detection_loss‚2.4337180733680723 0loss/FBt4/detection_loss‚0.36604840952680784 2loss/SRqqt1/assignment_loss‚1.7349106776641636 1loss/S"2Îó³Rqqt2/assignment_loss‚1.895401145626859 0loss/FBt3/assignment_loss‚0.2318414424856504 /loss/SRqqt2/detection_loss‚2.38346450067148 1loss/SRqqt4/detection_loss‚0.7845467212019672 /loss/FRt1/assignment_loss‚3.296915511290232 2loss/SRqqt4/assignment_loss‚0.8538918446382167 /loss/FRt1/detection_loss‚2.3201550549517074 /loss/FRt3/detection_loss‚2.2970834523439407 .loss/FBt3/detection_loss‚2.744440197944641 $loss/total_loss‚40.8911301445411 /loss/FRt4/detection_loss‚1.8494593926260006 1loss/SRqqt1/detection_loss‚2.1174946776250514 trainer/global_step‚10449 0loss/FRt3/assignment_loss‚2.5841456768305404 /loss/FRt4/assignment_loss‚1.093535225028577 _runtime‚8332.725393098 _step‚859 0loss/FRt2/assignment_loss‚3.4114104018491855 1loss/FBt2/assignment_loss‚0.46611560357268617 % _timestamp‚1.7884248329879904e+09 1loss/FBt4/assignment_loss‚0.07594950785033715 1loss/SRqqt3/detection_loss‚1.5066442155678357 1loss/SRqqt3/assignment_loss‚1.419357780167567 .loss/FRt2/detection_loss‚2.709893391412847 epoch‚8 0loss/FBt1/assignment_loss‚0.0699201366967625 /loss/FBt1/detection_loss‚2.2447871035999722úЀ_Þ&Ù /loss/FRt1/assignment_loss‚3.296915511290232 2loss/SRqqt4/assignment_loss‚0.8538918446382167 /loss/FRt1/detection_loss‚2.3201550549517074 /loss/FRt3/detection_loss‚2.2970834523439407 .loss/FBt3/detection_loss‚2.744440197944641 $loss/total_loss‚40.8911301445411 /loss/FRt4/detection_loss‚1.8494593926260006 1loss/SRqqt1/detection_loss‚2.1174946776250514 trainer/global_step‚10449 0loss/FRt3/assignment_loss‚2.5841456768305404 /loss/FRt4/assignment_loss‚1.093535225028577 _runtime‚8332.725393098 _step‚859 0loss/FRt2/assignment_loss‚3.4114104018491855 1loss/FBt2/assignment_loss‚0.46611560357268617 % _timestamp‚1.7884248329879904e+09 1loss/FBt4/assignment_loss‚0.07594950785033715 1loss/SRqqt3/detection_loss‚1.5066442155678357 1loss/SRqqt3/assignment_loss‚1.419357780167567 .loss/FRt2/detection_loss‚2.709893391412847 epoch‚8 0loss/FBt1/assignment_loss‚0.0699201366967625 /loss/FBt1/detection_loss‚2.2447871035999722 /loss/FBt2/detection_loss‚2.4337180733680723 0loss/FBt4/detection_loss‚0.36604840952680784 2loss/SRqqt1/assignment_loss‚1.7349106776641636 1loss/SRqqt2/assignment_loss‚1.895401145626859 0loss/FBt3/assignment_loss‚0.2318414424856504 /loss/SRqqt2/detection_loss‚2.38346450067148 1loss/SRqqt4/detection_loss‚0.7845467212019672ÛÕ \¨’ß&‚(:‡ ˆåäÔüþ˜~ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ç¡xgà&‚(:Ü ˆåä縷Πdisk.sda.in‚ 126.77734375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚35443.9658203125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 445.71875 disk.sda.out‚ 445.71875 disk.nvme0n1.in‚4.6875' proc.memory.availableMB‚ 951.0859375 proc.cpu.threads‚17 network.sent‚ 2.5246466e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375$ memory_percent‚94.19503211975098 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 cpu‚25.33946170422498$ disk./.usageGB‚260.7046928405762 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875" disk.dm-0.out‚35443.9658203125 disk.sda2.in‚0 network.recv‚ 3.039546e+06 disk.dm-1.out‚ 445.71875 disk.rbd1.out‚ 4.29296875' disk.nvme0c0n1.out‚35443.9658203125* proc.memory.percent‚16.921377182006836* disk./.usagePercent‚17.496110682338315 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0lQ¨#’á&‚(:‡ —åäÔÇñ~ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚123.112' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6224 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚< qâ&‚(:æ —åäÔÀ…š disk.dm-1.out‚ 446.14453125 disk.rbd1.out‚ 4.29296875$ disk./.usageGB‚260.7552146911621 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0$ memory_percent‚94.19355392456055 cpu‚25.3389429457865 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0 disk.drbd2265.out‚0& proc.memory.availableMB‚ 951.328125# disk.dm-0.out‚35500.55615234375 disk.sda.out‚ 446.14453125 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17 disk.dm-0.in‚4.6875* disk./.usagePercent‚17.499501246121792 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0& disk.nvme0n1.out‚35500.55615234375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35500.55615234375 disk.drbd2080.out‚0 disk.sda3.out‚ 446.14453125 disk.rbd1.in‚0 network.sent‚ 2.5354644e+07 network.recv‚ 3.045278e+06 disk.drbd1716.in‚00½v2ã&- _wandbruntime‚8368  _runtime‚8368ÖáÈä& & lr-AdamW/pg1‚8.743250563621295e-06 & lr-AdamW/pg2‚8.743250563621295e-06 trainer/global_step‚10499 % _timestamp‚1.7884248691969533e+09 _runtime‚8368.934487051 _step‚860vÌ1„Íå&Ç % _timestamp‚1.7884248691969533e+09 _runtime‚8368.934487051 _step‚860 & lr-AdamW/pg1‚8.743250563621295e-06 & lr-AdamW/pg2‚8.743250563621295e-06 trainer/global_step‚10499ܬ$¸©2æ&- _wandbruntime‚8368  _runtime‚8368F©3è'ç&âO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664452827648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619077064969728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139199168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619077739335168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610286154905088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10499 _step‚861 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198181076 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8790928067584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221706298368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50151452 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚603055674235052 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0*çÄí'è&çO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198161038 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9221706298368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619077064969728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50151452 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚603055674235052 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350782858 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152621820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89047718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89047716 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚675664452827648 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8791584430080 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619077064969728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152621822 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚666442770469376 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619077739335168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139199168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚610286136902144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚666442746529280 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152621820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619077739335168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚610286154905088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚10499 _step‚861 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198181076 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8790928067584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8790928067584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844Ý™ƒU2é&- _wandbruntime‚8369  _runtime‚8369L‰€Gê&ìM PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139212304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152635838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610342489940992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911815168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350815328 % _timestamp‚1.7884248698818564e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89055884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50156418 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726948760064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403871232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666504421207040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚603112066277900 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619134911815168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222551493120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_ 4bbytes.all.freed‚619134237436416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10499 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198199556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚862 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8369.619736097 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198199556 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8760036057612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727144699392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198179492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139211858 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50155974 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89055884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50156418 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚675726948760064 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8792403871232 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350835394 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350835394 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89055886 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚666504421207040 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚603112066277900 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚594352030220288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619134911815168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9222551493120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619134237436416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚610342489940992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 trainer/global_step‚10499 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚603111394800576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198199556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚862 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚8369.619736097 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198199556 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚610342507943936 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚610342507943936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8791747495424 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8760036057612 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350815328 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198179492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚675727144699392 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198179492 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139211858 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50155974 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152635838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚666504397266944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚610342489940992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619134911815168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350815328 % _timestamp‚1.7884248698818564e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚594352013180928 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944Þþß2ì&- _wandbruntime‚8369  _runtime‚8369Ç–#êYí&Ó /loss/FBt2/detection_loss‚2.6074667051434517 /loss/FRt1/assignment_loss‚2.785075413553338 0loss/FRt2/assignment_loss‚3.4091673838464835 0loss/SRqqt1/detection_loss‚1.926590541546995 trainer/global_step‚10499 1loss/SRqqt1/assignment_loss‚1.244177263865078 .loss/FBt3/detection_loss‚2.919770449399948 /loss/FBt4/detection_loss‚0.6957966854127172 /loss/FRt3/assignment_loss‚2.354616257548332 2loss/SRqqt4/assignment_loss‚0.8369395410316804 /loss/FBt2/assignment_loss‚0.423127230582395 -loss/FRt3/detection_loss‚2.36313090659678 &loss/total_loss‚44.168907060200375 1loss/FBt1/assignment_loss‚0.11925189344929095 /loss/FRt2/detection_loss‚2.7394421445695976 _runtime‚8369.620994762 % _timestamp‚1.7884248698835342e+09 _step‚863 0loss/FRt4/assignment_loss‚1.7903553505201597 2loss/SRqqt3/assignment_loss‚1.3550532169164538 2loss/FBt3/assignment_loss‚0.017244063357367168 1loss/SRqqt4/detection_loss‚0.8907924171478029 1loss/SRqqt2/assignment_loss‚2.080894878755013 1loss/FBt4/assignment_loss‚0.20665007743657582 .loss/FRt4/detection_loss‚2.562878192477935 0loss/SRqqt2/detection_loss‚2.729000801841418 .loss/FBt1/detection_loss‚4.346001207828522 epoch‚8 /loss/FRt1/detection_loss‚2.1401498474572835 1loss/SRqqt3/detection_loss‚1.6253345899157605iE¸±^î&Ø % _timestamp‚1.7884248698835342e+09 _step‚863 0loss/FRt4/assignment_loss‚1.7903553505201597 2loss/SRqqt3/assignment_loss‚1.3550532169164538 2loss/FBt3/assignment_loss‚0.017244063357367168 1loss/SRqqt4/detection_loss‚0.8907924171478029 1loss/SRqqt2/assignment_loss‚2.080894878755013 1loss/FBt4/assignment_loss‚0.20665007743657582 .loss/FRt4/detection_loss‚2.562878192477935 0loss/SRqqt2/detection_loss‚2.729000801841418 .loss/FBt1/detection_loss‚4.346001207828522 epoch‚8 /loss/FRt1/detection_loss‚2.1401498474572835 1loss/SRqqt3/detection_loss‚1.6253345899157605 /loss/FBt2/detection_loss‚2.6074667051434517 /loss/FRt1/assignment_loss‚2.785075413553338 0loss/FRt2/assignment_loss‚3.4091673838464835 0loss/SRqqt1/detection_loss‚1.926590541546995 trainer/global_step‚10499 1loss/SRqqt1/assignment_loss‚1.244177263865078 .loss/FBt3/detection_loss‚2.919770449399948 /loss/FBt4/detection_loss‚0.6957966854127172 /loss/FRt3/assignment_loss‚2.354616257548332 2loss/SRqqt4/assignment_loss‚0.8369395410316804 /loss/FBt2/assignment_loss‚0.423127230582395 -loss/FRt3/detection_loss‚2.36313090659678 &loss/total_loss‚44.168907060200375 1loss/FBt1/assignment_loss‚0.11925189344929095 /loss/FRt2/detection_loss‚2.7394421445695976 _runtime‚8369.620994762ß:9ñÞ’ï&‚(:‡ ¦åäԇυ~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.906' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëTj–tð&‚(:é ¦åäÔ­†ó€ disk.sda2.out‚0 network.recv‚ 3.049929e+06 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-1.out‚ 446.80859375 disk.nvme0n1.in‚4.6875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚35562.26318359375 disk.sda.out‚ 446.80859375 disk.rbd0.in‚0 disk.drbd1716.out‚0$ memory_percent‚94.19341087341309' proc.memory.availableMB‚ 951.3515625" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836) disk./.usagePercent‚17.50009825569164 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda3.out‚ 446.80859375 disk.drbd2265.out‚0 cpu‚25.338428657665045 network.sent‚ 2.5360657e+07% disk./.usageGB‚260.76411056518555 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚35562.26318359375 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375& disk.nvme0n1.out‚35562.26318359375 disk.sda3.in‚ 126.77734375 disk.drbd1716.in‚0 {kbñ&‚(:… µåäÔêä~ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡™ûìsò&‚(:è µåäÔŇŽ" proc.memory.rssMB‚ 2772.3984375 cpu‚25.337912860194592 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.921377182006836 disk.drbd2266.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚35602.11865234375 disk.rbd0.out‚0 disk.drbd1716.out‚0# disk.dm-0.out‚35602.11865234375 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.29296875$ memory_percent‚94.19376850128174 proc.cpu.threads‚17( disk.nvme0c0n1.out‚35602.11865234375 disk.sda3.out‚ 447.36328125$ disk./.usageGB‚260.7778205871582 disk.drbd1716.in‚0 disk.sda.out‚ 447.36328125( proc.memory.availableMB‚ 951.29296875 disk.dm-1.out‚ 447.36328125 network.sent‚ 2.5469296e+07 network.recv‚ 3.055891e+06( disk./.usagePercent‚17.5010183467697 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0CÂó&‚(:’ ÄåäÔ©Áª~ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.078' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.215600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—¿XÇkô&‚(:à ÄåäÔòæ’ disk.sda1.out‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚35650.498046875 disk.sda3.in‚ 126.77734375 network.sent‚ 2.5475599e+07 network.recv‚ 3.060422e+06! disk.dm-0.out‚35650.498046875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375% disk./.usageGB‚260.77917098999023 disk.drbd2265.in‚0$ disk.nvme0n1.out‚35650.498046875( proc.memory.availableMB‚ 951.29296875 disk.dm-1.out‚ 447.8984375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.29296875 disk.sda.out‚ 447.8984375 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.nvme0n1.in‚4.6875 disk.sda2.in‚0 disk.sda3.out‚ 447.8984375* proc.memory.percent‚16.921377182006836) disk./.usagePercent‚17.50110897343682 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875$ memory_percent‚94.19376850128174" proc.memory.rssMB‚ 2772.3984375 cpu‚25.33740011331603 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17 disk.dm-0.in‚4.6875 disk.rbd1.in‚00s2õ&- _wandbruntime‚8405  _runtime‚8405îH1Èö& & lr-AdamW/pg1‚8.211299858746428e-06 & lr-AdamW/pg2‚8.211299858746428e-06 trainer/global_step‚10549 % _timestamp‚1.7884249057670884e+09 _runtime‚8405.504613881 _step‚864ð±ÒÍ÷&Ç & lr-AdamW/pg2‚8.211299858746428e-06 trainer/global_step‚10549 % _timestamp‚1.7884249057670884e+09 _runtime‚8405.504613881 _step‚864 & lr-AdamW/pg1‚8.211299858746428e-06ಋà 2ø&- _wandbruntime‚8405  _runtime‚84054:%;ç'ù&áO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚10549 _runtime‚8405.505601144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 $ _timestamp‚1.788424905767855e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139834334 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790238281728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597042315578368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139834777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚613103788844544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50378218 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790063300608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702717952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872Ä3ŠVú&æO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 139834777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_staÓE/ñ–%rt/active_bytes.large_pool.freed‚613103788844544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚605842387579576 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153322620 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621935702717952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚613103806847488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚441 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199087090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8832570235904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50378218 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678790063300608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621935702717952 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8832570235904 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352429750 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199107128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚10549 _runtime‚8405.505601144 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199087090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚597042332617728 $ _timestamp‚1.788424905767855e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚605841716102460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139834334 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8800054961848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613103788844544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89456118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚669525283412480 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678790238281728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚597042315578368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153322622 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚669525307352576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0á¶àúq2û&- _wandbruntime‚8406  _runtime‚8406§7 ·ò&ü&ìM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139847438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884249064692836e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8832733301248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852560627712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389677056 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153336636 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621992875184640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8406.207313736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265796379648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265626477568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605897436667984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50382706 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139846990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚866 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153336636 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50383152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605898108145308 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678852754469888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733301248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597097236748288?ã÷&ý&ñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613160159886336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613160141883392 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9265796379648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352462246 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613160141883392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9265626477568 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚605897436667984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153336638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50382706 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8800871397020 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 352442180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚597097219708928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139846990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚866 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153336636 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50383152 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199125608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352462246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153336638 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605898108145308 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚621993549563392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169902080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89464286 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199105544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚678852754469888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8800216959056 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8832733301248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚597097236748288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89464284 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚669586934150144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352442180 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139847438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 % _timestamp‚1.7884249064692836e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8832733301248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678852560627712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚669586958090240 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199105544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8833389677056 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚613160159886336 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8833389677056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153336636 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚621992875184640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 _runtime‚8406.207313736â*yQ®2þ&- _wandbruntime‚8406  _runtime‚8406¶_á]ÿ&× 1loss/FBt2/assignment_loss‚0.27601609432349034 .loss/FBt3/detection_loss‚2.680695193154471 trainer/global_step‚10549 _runtime‚8406.208450561 2loss/SRqqt4/assignment_loss‚1.0522259599030284 1loss/SRqqt4/detection_loss‚1.1067078466793256 .loss/FBt1/detection_loss‚2.936022616349734 /loss/FRt3/detection_loss‚2.5278293460141867 1loss/SRqqt2/detection_loss‚2.4246090345266387 %loss/total_loss‚42.79051249328228 _step‚867 /loss/FRt1/detection_loss‚2.1786417326201564 /loss/FRt4/detection_loss‚1.4067245831092199 epoch‚8 0loss/FRt1/assignment_loss‚3.6301851337370663 /loss/FRt3/assignment_loss‚2.851815818498532 1loss/SRqqt2/assignment_loss‚1.676853042005039 /loss/FRt2/assignment_loss‚3.465577178531223 1loss/SRqqt3/assignment_loss‚1.276514512810351 0loss/FBt1/assignment_loss‚0.4489694999244351 1loss/FBt3/assignment_loss‚0.19496718091457815 1loss/FBt4/assignment_loss‚0.16168410188097962 /loss/FRt2/detection_loss‚2.9332119954956903 1loss/SRqqt3/detection_loss‚1.5483743287622929 0loss/FRt4/assignment_loss‚1.4884209652741751 2loss/SRqqt1/assignment_loss‚1.6806422480663588 1loss/SRqqt1/detection_loss‚1.7182051092386246 /loss/FBt2/detection_loss‚2.5023063520590463 /loss/FBt4/detection_loss‚0.6233126194036397 % _timestamp‚1.7884249064710095e+09.@ŽŠb€'Ü /loss/FRt1/detection_loss‚2.1786417326201564 /loss/FRt4/detection_loss‚1.4067245831092199 epoch‚8 0loss/FRt1/assignment_loss‚3.6301851337370663 /loss/FRt3/assignment_loss‚2.851815818498532 1loss/SRqqt2/assignment_loss‚1.676853042005039 /loss/FRt2/assignment_loss‚3.465577178531223 1loss/SRqqt3/assignment_loss‚1.276514512810351 0loss/FBt1/assignment_loss‚0.4489694999244351 1loss/FBt3/assignment_loss‚0.19496718091457815 1loss/FBt4/assignment_loss‚0.16168410188097962 /loss/FRt2/detection_loss‚2.9332119954956903 1loss/SRqqt3/detection_loss‚1.5483743287622929 0loss/FRt4/assignment_loss‚1.4884209652741751 2loss/SRqqt1/assignment_loss‚1.6806422480663588 1loss/SRqqt1/detection_loss‚1.7182051092386246 /loss/FBt2/detection_loss‚2.5023063520590463 /loss/FBt4/detection_loss‚0.6233126194036397 % _timestamp‚1.7884249064710095e+09 1loss/FBt2/assignment_loss‚0.27601609432349034 .loss/FBt3/detection_loss‚2.680695193154471 trainer/global_step‚10549 _runtime‚8406.208450561 2loss/SRqqt4/assignment_loss‚1.0522259599030284 1loss/SRqqt4/detection_loss‚1.1067078466793256 .loss/FBt1/detection_loss‚2.936022616349734 /loss/FRt3/detection_loss‚2.5278293460141867 1loss/SRqqt2/detection_loss‚2.4246090345266387 %loss/total_loss‚42.79051249328228 _step‚867ã€ù_Æ&'‚(:‡ ÓåäԺЋ~ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚186.523' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.3046 gpu.0.smCloc»h]Ílk‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ògh¡k‚'‚(:à ÓåäÔÿüŽ€ disk.rbd1.in‚0 disk.sda.in‚ 126.77734375& disk.nvme0c0n1.out‚35693.458984375 disk.drbd2080.in‚0 disk.dm-1.out‚ 448.1796875 disk.drbd2266.out‚0 cpu‚25.336866233809737 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚ 448.1796875 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375$ memory_percent‚94.19400691986084 disk.sda.out‚ 448.1796875 disk.nvme0n1.in‚4.6875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0( proc.memory.availableMB‚ 951.25390625 network.recv‚ 3.063819e+06! disk.dm-0.out‚35693.458984375$ disk.nvme0n1.out‚35693.458984375 network.sent‚ 2.5480919e+07 disk.rbd0.out‚0$ disk./.usageGB‚260.7907829284668 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0) disk./.usagePercent‚17.50188826037105 disk.rbd1.out‚ 4.29296875 disk.drbd1716.out‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.sda1.in‚0* proc.memory.percent‚16.921377182006836né3ƒ'‚(:’ âåäÔÒÖµ~ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.696' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.139199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚031_„'‚(:Ô âåäÔ˜£Ã€ disk.sda2.out‚0( proc.memory.availableMB‚ 951.23046875* proc.memory.percent‚16.921377182006836 disk.drbd2265.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚35737.4541015625 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚448.75 disk.dm-1.out‚448.75 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚35737.4541015625 disk.drbd2266.out‚0* disk./.usagePercent‚17.500627423264916$ disk./.usageGB‚260.7719955444336 disk.drbd2265.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2772.3984375 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 3.070087e+06 disk.nvme0n1.in‚4.6875 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375% disk.nvme0n1.out‚35737.4541015625# memory_percent‚94.1941499710083 cpu‚25.336384401394422 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.29296875 network.sent‚ 2.5589593e+07 disk.sda.out‚448.75’xV:2…'- _wandbruntime‚8442  _runtime‚8442Äy¼@Ȇ' _runtime‚8442.417076019 _step‚868 & lr-AdamW/pg1‚7.695214949298185e-06 & lr-AdamW/pg2‚7.695214949298185e-06 trainer/global_step‚10599 % _timestamp‚1.7884249426795328e+09¼}x͇'Ç & lr-AdamW/pg1‚7.695214949298185e-06 & lr-AdamW/pg2‚7.695214949298185e-06 trainer/global_step‚10599 % _timestamp‚1.7884249426795328e+09 _runtime‚8442.417076019 _step‚868äŒD2ˆ'- _wandbruntime‚8442  _runtime‚844201o´è'‰'âO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89864518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672607844235776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599787539145728 trainer/global_step‚10599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556041728 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969868 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624795014831616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200013142 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872899679232Vtfí'Š'çO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚608627757969868 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 354036562 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚624795014831616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200013142 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚672607820295680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154023422 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8872899679232 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89864518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚672607844235776 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚624794340466176 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599787539145728 trainer/global_step‚10599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615921440786944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8873556041728å´ðã2‹'- _wandbruntime‚8443  _runtime‚8443ñ ”…Ù"Œ'ìM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚624852187311616 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚615977811828736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9308624707584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154037438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚681978095740928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚599842443276288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50610886 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚672669471033344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚599842426236928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140484013 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚870 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200031596 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187311616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375482880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978293777408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154037436 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140483570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719107072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 % _timestamp‚1.7884249433739548e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672669494973440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8841706736428 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚8443.111981885 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615977811828736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200051660 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140484013 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8841052298464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚870 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200051660 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200031596 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚624852187311616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8874375482880 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681978293777408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615977793825792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154037436 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140483570 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89872686 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8873719107072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615977793825792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 % _timestamp‚1.7884249433739548e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354069032 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8873719107072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚672669494973440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8841706736428 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 _runtime‚8443.111981885 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154037438 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89872684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154037436 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚443 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚624851512932864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615977811828736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354089098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚608684150012716 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 354069032 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141108407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚618739092729344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8881071202908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚618739110732288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50835489 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141107951 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350874045440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚8479.252800063 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939194 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685041427176960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675690381118976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8881725640664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884249795151372e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚873 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885485056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541847552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200959232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚618739092729344 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627652978214400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355683454 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚618739092729344 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚618739110732288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚611413799837276 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141108407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚618739092729344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8881071202908 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚618739110732288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50835489 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141107951 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 195952640 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602532728634368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9350874045440 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚8479.252800063 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200939194 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ^JDeviceStatsMonitorq˜E G.on_train_batch_start/inactive_split_bytes.all.allocated‚685041427176960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172012544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90272918 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚675690381118976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154724220 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200959232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8881725640664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884249795151372e+09 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚456 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90272916 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚627652978214400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚873 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 154724220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675690357178880 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355663414 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154724222 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚611414471314392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8913885485056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8914541847552 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚602532745673728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚627653652579840 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8913885485056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8914541847552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944é!Zö2'- _wandbruntime‚8479  _runtime‚8479 iXÁò&ž'ìM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚627710825059840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355695884 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8914704912896 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141120642 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8479.943153571 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103744748032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825059840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚618795463771136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695884 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚874 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚618795463771136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351736831488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150681088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8881887637872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675752031856640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8882542075836 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90281086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50839558 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8914704912896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685103930201600 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355715950 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200977712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8914704912896 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685103930201600 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141120642 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 _runtime‚8479.943153571 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8915361288704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685103744748032 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚627710825059840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚618795463771136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚602587632764928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚602587649804288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚675752007916544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 355715950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355695884 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚874 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚618795463771136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200977712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90281084 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9351736831488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚618795445768192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚618795445768192 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚627710150681088 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154738236 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200957648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8881887637872 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889êeÌ`ú2 '- _wandbruntime‚8479  _runtime‚8479úo½Ð`¡'Ú 0loss/FRt4/assignment_loss‚1.4228907934689925 1loss/FBt1/assignment_loss‚0.42767031550068746 _runtime‚8479.944523446 1loss/FBt2/assignment_loss‚0.31750925156732346 %loss/total_loss‚41.28572651058276 2loss/SRqqt1/assignment_loss‚1.6810648827148336 /loss/FBt1/detection_loss‚2.3302059986374597 _step‚875 0loss/FRt2/assignment_loss‚2.7734301501306993 /loss/FRt2/detection_loss‚2.9814697997323396 /loss/FRt4/detection_loss‚1.4810825360275932 0loss/SRqqt1/detection_loss‚1.763591182171082 0loss/SRqqt3/detection_loss‚1.757740817964077 /loss/FBt2/detection_loss‚2.8503260070627388 2loss/SRqqt3/assignment_loss‚1.4225361108034849 0loss/FBt4/assignment_loss‚0.2156122740230872 .loss/FRt3/detection_loss‚2.365248397986094 1loss/SRqqt2/detection_loss‚3.0564504663149514 1loss/SRqqt4/detection_loss‚0.8695646007300207 trainer/global_step‚10649 2loss/SRqqt2/assignment_loss‚1.1164843142032623 2loss/SRqqt4/assignment_loss‚0.8042323601946822 2loss/FBt3/assignment_loss‚0.046784860500338254 /loss/FRt1/detection_loss‚2.2674174706141152 epoch‚8 0loss/FRt1/assignment_loss‚2.9133763591448467 /loss/FRt3/assignment_loss‚2.785435606373681 /loss/FBt3/detection_loss‚2.9310905933380127 /loss/FBt4/detection_loss‚0.7045113613783587 % _timestamp‚1.7884249802067971e+09b¯ˆfe¢'ß 2loss/FBt3/assignment_loss‚0.046784860500338254 /loss/FRt1/detection_loss‚2.2674174706141152 epoch‚8 0loss/FRt1/assignment_loss‚2.9133763591448467 /loss/FRt3/assignment_loss‚2.785435606373681 /loss/FBt3/detection_loss‚2.9310905933380127 /loss/FBt4/detection_loss‚0.7045113613783587 % _timestamp‚1.7884249802067971e+09 0loss/FRt4/assignment_loss‚1.4228907934689925 1loss/FBt1/assignment_loss‚0.42767031550068746 _runtime‚8479.944523446 1loss/FBt2/assignment_loss‚0.31750925156732346 %loss/total_loss‚41.28572651058276 2loss/SRqqt1/assignment_loss‚1.6810648827148336 /loss/FBt1/detection_loss‚2.3302059986374597 _step‚875 0loss/FRt2/assignment_loss‚2.7734301501306993 /loss/FRt2/detection_loss‚2.9814697997323396 /loss/FRt4/detection_loss‚1.4810825360275932 0loss/SRqqt1/detection_loss‚1.763591182171082 0loss/SRqqt3/detection_loss‚1.757740817964077 /loss/FBt2/detection_loss‚2.8503260070627388 2loss/SRqqt3/assignment_loss‚1.4225361108034849 0loss/FBt4/assignment_loss‚0.2156122740230872 .loss/FRt3/detection_loss‚2.365248397986094 1loss/SRqqt2/detection_loss‚3.0564504663149514 1loss/SRqqt4/detection_loss‚0.8695646007300207 trainer/global_step‚10649 2loss/SRqqt2/assignment_loss‚1.1164843142032623 2loss/SRqqt4/assignment_loss‚0.8042323601946822릒£'‚(:‡ žæäÔ´ë²~ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ã¸d¤'‚(:Ù žæäÔ½—ê€ disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚450.9375 disk.drbd2265.in‚0 disk.rbd1.out‚ 4.29296875$ memory_percent‚94.20008659362793 network.recv‚ 3.088557e+06 disk.sda1.out‚0 disk.sda.out‚450.9375" proc.memory.rssMB‚ 2772.3984375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875' proc.memory.availableMB‚ 950.2578125 cpu‚25.33433542392637 disk.drbd2266.out‚0$ disk./.usageGB‚260.8395576477051% disk.nvme0n1.out‚35914.5087890625 proc.cpu.threads‚17 network.sent‚ 2.5715922e+07 disk.drbd2265.out‚0 disk.dm-1.out‚450.9375 disk.sda.in‚ 126.77734375* proc.memory.percent‚16.921377182006836 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0' disk.nvme0c0n1.out‚35914.5087890625" disk.dm-0.out‚35914.5087890625* disk./.usagePercent‚17.505161572703845 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0`¤‹ª¥'‚(:’ ­æäÔÖž¯~ gpu.0.gpu‚30.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.195800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðfúk¦'‚(:à ­æäÔž„ž€ disk.sda.in‚ 126.77734375 disk.sda1.in‚0# memory_percent‚94.1986083984375 disk.sda2.out‚0% disk.nvme0n1.out‚35959.2470703125 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875* disk./.usagePercent‚17.506253700844915 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.out‚ 451.52734375* proc.memory.percent‚16.921377182006836 cpu‚25.333809893173356 disk.sda2.in‚0 network.recv‚ 3.094383e+06 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 451.52734375 disk.rbd1.in‚0% disk./.usageGB‚260.85583114624023 disk.drbd1716.out‚0" disk.dm-0.out‚35959.2470703125 disk.dm-1.in‚ 126.77734375! proc.memory.availableMB‚950.5 network.sent‚ 2.5824528e+07 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2772.3984375 disk.sda3.out‚ 451.52734375' disk.nvme0c0n1.out‚35959.2470703125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.out‚ 4.29296875™Ì§º2§'- _wandbruntime‚8516  _runtime‚8516Å Ÿž¨'¾ % _timestamp‚1.788Ýv¨4250163904972e+09 _runtime‚8516.128159536 _step‚876 $ lr-AdamW/pg1‚6.7110693931465e-06 $ lr-AdamW/pg2‚6.7110693931465e-06 trainer/global_step‚10699Àê³É©'à % _timestamp‚1.7884250163904972e+09 _runtime‚8516.128159536 _step‚876 $ lr-AdamW/pg1‚6.7110693931465e-06 $ lr-AdamW/pg2‚6.7110693931465e-06 trainer/global_step‚10699ì8ëÄ2ª'- _wandbruntime‚8516  _runtime‚8516‘‹öè'«'âO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚877 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884250163913422e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290328064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141744035 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155425022 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8955527653376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155425020 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630511615962624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172883968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90681318 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688166687195136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚678772894062080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871290880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9393793133056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 148943872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51062719 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51063167 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _runtime‚8516.129086866oí'¬'çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51063167 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚614199841704684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201885284 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _runtime‚8516.129086866 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚678772918002176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚614200513181800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 _step‚877 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884250163913422e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630512290328064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630512290328064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155425020 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621556744671744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 141744035 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155425022 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8955527653376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155425020 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630511615962624 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 172883968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚605277935162368 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155425022 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357310306 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90681318 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚688166687195136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚678772894062080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621556744671744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8954871290880 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9393793133056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90681316 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 148943872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 201865246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630511615962624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621556762674688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51062719 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844íàùÞ2­'- _wandbruntime‚8516  _runtime‚8516ò6Öñ&®'ëM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90689486 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462808064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605332856332288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141756718 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51067234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51067680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚678834568739840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462808064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8516.814792814 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688229175984128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8956347094528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621613115713536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347094528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229374020608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9394805280768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415244 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚688229175984128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357322736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8956347094528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621613115713536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8956347094528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201903764 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚688229374020608 TFDeviceStatsMonitor.on_train_batchAzl·É_end/inactive_split_bytes.all.current‚ 198036480 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9394805280768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8923377415244 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357342802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90689486 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 _step‚878 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚630569462808064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚605332856332288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141756718 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8955690718720 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51067234 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90689484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201883700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51067680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚678834568739840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚630569462808064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8516.814792814 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155439038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚10699 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414îÑr82°'- _wandbruntime‚8516  _runtime‚8516[78z[±'Õ 2loss/SRqqt3/assignment_loss‚1.3484275050906707 2loss/SRqqt4/assignment_loss‚1.0835367899486108 .loss/FBt3/detection_loss‚2.693142226764134 0loss/FRt1/assignment_loss‚3.3436473194629914 2loss/SRqqt2/assignment_loss‚1.6085203232673497 1loss/FBt2/assignment_loss‚0.16797993387807253 1loss/FBt3/assignment_loss‚0.28724510565539874 1loss/SRqqt3/detection_loss‚1.6548688534809195 .loss/FBt2/detection_loss‚3.042704224586487 _runtime‚8516.815925562 trainer/global_step‚10699 .loss/FRt2/assignment_loss‚3.51069198846817 1loss/FBt4/assignment_loss‚0.22834327280558372 /loss/FBt4/detection_loss‚0.4622630208777787 % _timestamp‚1.7884250170784624e+09 1loss/FBt1/assignment_loss‚0.27056441652322455 .loss/FRt3/detection_loss‚2.584467633627355 0loss/SRqqt1/detection_loss‚1.668750661611557 /loss/FRt2/detection_loss‚2.7694159848349433 0loss/FRt4/assignment_loss‚1.7122141558390398 .loss/FRt1/detection_loss‚2.117504658237581 1loss/SRqqt2/detection_loss‚2.7599626825406003 /loss/FBt1/detection_loss‚1.8378774608884538 /loss/FRt3/assignment_loss‚2.239507335424423 2loss/SRqqt1/assignment_loss‚1.3508458106964827 .loss/FRt4/detection_loss‚1.825740765995131 0loss/SRqqt4/detection_loss‚1.275168200497303 &loss/total_loss‚41.843390331002254 epoch‚8 _step‚879ôËiÇ`²'Ú 2loss/SRqqt3/assignment_loss‚1.3484275050906707 2loss/SRqqt4/assignment_loss‚1.0835367899486108 .loss/FBt3/detection_loss‚2.693142226764134 0loss/FRt1/assignment_loss‚3.3436473194629914 2loss/SRqqt2/assignment_loss‚1.6085203232673497 1loss/FBt2/assignment_loss‚0.16797993387807253 1loss/FBt3/assignment_loss‚0.28724510565539874 1loss/SRqqt3/detection_loss‚1.6548688534809195 .loss/FBt2/detection_loss‚3.042704224586487 _runtime‚8516.815925562 trainer/global_step‚10699 .loss/FRt2/assignment_loss‚3.51069198846817 1loss/FBt4/assignment_loss‚0.22834327280558372 /loss/FBt4/detection_loss‚0.4622630208777787 % _timestamp‚1.7884250170784624e+09 1loss/FBt1/assignment_loss‚0.27056441652322455 .loss/FRt3/detection_loss‚2.584467633627355 0loss/SRqqt1/detection_loss‚1.668750661611557 /loss/FRt2/detection_loss‚2.7694159848349433 0loss/FRt4/assignment_loss‚1.7122141558390398 .loss/FRt1/detection_loss‚2.117504658237581 1loss/SRqqt2/detection_loss‚2.7599626825406003 /loss/FBt1/detection_loss‚1.8378774608884538 /loss/FRt3/assignment_loss‚2.239507335424423 2loss/SRqqt1/assignment_loss‚1.3508458106964827 .loss/FRt4/detection_loss‚1.825740765995131 0loss/SRqqt4/detection_loss‚1.275168200497303 &loss/total_loss‚41.843390331002254 epoch‚8 _step‚879ïÍwÕ’³'‚(:‡ ¼æäÔéÂË gpu.0.gpu‚26.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.0026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hAãg´'‚(:Ü ¼æäÔ«–Ä€ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 950.93359375 network.recv‚ 3.099351e+06* disk./.usagePercent‚17.507183520209278 disk.nvme0n1.in‚4.6875 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda2.out‚0 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.sda.out‚ 451.99609375 disk.sda1.out‚0 disk.sda3.out‚ 451.99609375 cpu‚25.33333914485935 network.sent‚ 2.5830911e+07 disk.drbd1716.in‚0 disk.drbd1716.out‚0" disk.nvme0n1.out‚ 36014.3359375 disk.drbd2265.out‚0 disk.dm-0.out‚ 36014.3359375 disk.rbd1.out‚ 4.29296875$ memory_percent‚94.19596195220947 disk.drbd2080.in‚0* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17# disk./.usageGB‚260.869686126709 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875$ disk.nvme0c0n1.out‚ 36014.3359375 disk.dm-1.out‚ 451.99609375 disk.drbd2266.in‚0 disk.sda1.in‚0õ—¹1’µ'‚(:‡ ËæäÔ«» ~ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.849' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±]àž_¶'‚(:Ô ËæäÔ¸íÍ€ disk.drbd1716.in‚0# disk.nvme0n1.out‚36053.76953125 disk.drbd2080.out‚0) disk./.usagePercent‚17.50776798540995 disk.sda2.in‚0 disk.rbd1.out‚ 4.29296875 disk.sda1.out‚0% disk.nvme0c0n1.out‚36053.76953125' proc.memory.availableMB‚ 950.4296875 proc.cpu.threads‚17 network.sent‚ 2.5938891e+07 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 452.59375 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 452.59375 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.6875$ disk./.usageGB‚260.8783950805664 disk.sda.out‚ 452.59375" proc.memory.rssMB‚ 2772.3984375 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375* proc.memory.percent‚16.921377182006836 cpu‚25.33284228108777 network.recv‚ 3.10487e+06 disk.sda.in‚ 126.77734375 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-0.out‚36053.76953125$ memory_percent‚94.19903755187988 disk.sda1.in‚0Íbÿe’·'‚(:‡ ÚæäÔÊ˸~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”y¼.m¸'‚(:â ÚæäÔʽ¤€ disk.drbd2266.out‚0 disk.sda.out‚ 453.359375& disk.nvme0n1.out‚36101.22802734375( disk.nvme0c0n1.out‚36101.22802734375 disk.drbd2080.in‚0 disk.rbd1.in‚0$ memory_percent‚94.19901371002197 disk.drbd1716.out‚0 disk.sda2.out‚0 cpu‚25.33234954081489 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0# disk.dm-0.out‚36101.22802734375) disk./.usagePercent‚17.50882939264691 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚ 453.359375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.recv‚ 3.109352e+06$ disk./.usageGB‚260.8942108154297 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875 disk.drbd2266.in‚0( proc.memory.availableMB‚ 950.43359375" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5944945e+07 disk.drbd1716.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.29296875 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 453.359375í}Ø)2¹'- _wandbruntime‚8552  _runtime‚8552mÌÌ`Ⱥ' _step‚880 & lr-AdamW/pg1‚6.243213840901901e-06 & lr-AdamW/pg2‚6.243213840901901e-06 trainer/global_step‚10749 % _timestamp‚1.7884250529595554e+09 _runtime‚8552.697076653ÔȾ£Í»'Ç _step‚880 & lr-AdamW/pg1‚6.243213840901901e-06 & lr-AdamW/pg2‚6.243213840901901e-06 trainer/global_step‚10749 % _timestamp‚1.7884250529595554e+09 _runtime‚8552.697076653ðqäsö2¼'- _wandbruntime‚8552  _runtime‚8552x®5è'½'âO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142380953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚681855430945280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142380497 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚608023158729728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292307125760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202791298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624374414617088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633370253710848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633370928076288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91089716 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436690713600 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51291235 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292121658880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9436852240384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8995857096704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51290781 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202811336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 _step‚881 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8996513459200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128ãQ¾'çO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142380953 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚681855430945280 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8962741881724 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142380497 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚608023158729728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚691292307125760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616985883572092 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202791298 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚624374414617088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8996513459200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚633370253710848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚454 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚624374396614144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚633370928076288 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91089716 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8995857096704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚633370253710848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9436690713600 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/ŒÝE Ýallocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51291235 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚691292121658880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9436852240384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616986555049208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8995857096704 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358937158 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51290781 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 202811336 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚633370928076288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202811336 _step‚881 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚624374414617088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8996513459200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8963396319480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156125820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908ñÕ˜Š®2¿'- _wandbruntime‚8553  _runtime‚8553ÏX&ò&À'ìM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚617041604137616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691354821080576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691354631432704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156139836 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91097884 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142393585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437549749760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437715457536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332900352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142393121 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614940 _runtime‚8553.393176863 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676524544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633428100556288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Yð ]÷&Á'ñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10749 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 202809752 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚624430749652992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9437715457536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8963558316688 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8997332900352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358969654 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142393121 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚617042275614940 _runtime‚8553.393176863 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358949588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8996676524544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358949588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 189647872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156139836 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚633428100556288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358969654 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8996676524544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚617041604137616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚691354821080576 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 202829816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚691354631432704 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚624430749652992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 202829816 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156139838 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156139836 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 165707776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8964212754652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91097884 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142393585 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚681917105623040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9437549749760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚633427426177536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156139838 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202809752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91097886 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚624430767655936 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚681917081682944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚608078045820928ò¦u­2Â'- _wandbruntime‚8553  _runtime‚8553H{|E[Ã'Õ 1loss/SRqqt2/assignment_loss‚2.027031299897603 /loss/FBt2/detection_loss‚2.5156421936475315 % _timestamp‚1.7884250536567326e+09 _runtime‚8553.394459193 0loss/FRt1/assignment_loss‚3.2762700560001226 /loss/FRt2/assignment_loss‚2.772932444016139 0loss/FRt3/assignment_loss‚2.7439600151518118 0loss/FBt1/assignment_loss‚0.2951105657266453 0loss/SRqqt4/detection_loss‚1.033867997486913 $loss/total_loss‚41.0947436622821 0loss/FRt4/assignment_loss‚0.9051887237507364 /loss/FRt1/detection_loss‚2.1187034788040013 /loss/FRt4/detection_loss‚1.8203882242188505 0loss/SRqqt3/detection_loss‚1.647443892310063 _step‚883 2loss/SRqqt3/assignment_loss‚1.2992107303192217 -loss/FBt3/detection_loss‚2.49132888764143 epoch‚8 trainer/global_step‚10749 /loss/FRt2/detection_loss‚3.0154957731564838 /loss/FRt3/detection_loss‚2.3392526024709577 1loss/FBt2/assignment_loss‚0.39970952017063155 /loss/FBt1/detection_loss‚2.3261169232428074 1loss/FBt3/assignment_loss‚0.44217735560960136 1loss/FBt4/assignment_loss‚0.14313634310186352 1loss/SRqqt1/detection_loss‚1.9722782328076984 1loss/SRqqt2/detection_loss‚2.3421542763710024 2loss/SRqqt1/assignment_loss‚1.6493601798201385 1loss/SRqqt4/assignment_loss‚0.921770981483398 /loss/FBt4/detection_loss‚0.5962129650764506L!›Ñ`Ä'Ú /loss/FRt4/detection_loss‚1.8203882242188505 0loss/SRqqt3/detection_loss‚1.647443892310063 _step‚883 2loss/SRqqt3/assignment_loss‚1.2992107303192217 -loss/FBt3/detection_loss‚2.49132888764143 epoch‚8 trainer/global_step‚10749 /loss/FRt2/detection_loss‚3.0154957731564838 /loss/FRt3/detection_loss‚2.3392526024709577 1loss/FBt2/assignment_loss‚0.39970952017063155 /loss/FBt1/detection_loss‚2.3261169232428074 1loss/FBt3/assignment_loss‚0.44217735560960136 1loss/FBt4/assignment_loss‚0.14313634310186352 1loss/SRqqt1/detection_loss‚1.9722782328076984 1loss/SRqqt2/detection_loss‚2.3421542763710024 2loss/SRqqt1/assignment_loss‚1.6493601798201385 1loss/SRqqt4/assignment_loss‚0.921770981483398 /loss/FBt4/detection_loss‚0.5962129650764506 1loss/SRqqt2/assignment_loss‚2.027031299897603 /loss/FBt2/detection_loss‚2.5156421936475315 % _timestamp‚1.7884250536567326e+09 _runtime‚8553.394459193 0loss/FRt1/assignment_loss‚3.2762700560001226 /loss/FRt2/assignment_loss‚2.772932444016139 0loss/FRt3/assignment_loss‚2.7439600151518118 0loss/FBt1/assignment_loss‚0.2951105657266453 0loss/SRqqt4/detection_loss‚1.033867997486913 $loss/total_loss‚41.0947436622821 0loss/FRt4/assignment_loss‚0.9051887237507364 /loss/FRt1/detection_loss‚2.1187034788040013óÒ(‰Å'‚(:’ éæäÔŠ·‡~ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.425' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.885000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F*‘‘uÆ'‚(:ê éæäÔßö disk.drbd2266.out‚0 disk.sda.out‚ 453.87109375 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚94.19772624969482" proc.memory.rssMB‚ 2772.3984375$ disk./.usageGB‚260.9048728942871 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 950.64453125* proc.memory.percent‚16.921377182006836 cpu‚25.331852143948183 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚36138.90380859375 network.recv‚ 3.113308e+06* disk./.usagePercent‚17.509544933705108 disk.drbd2266.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚36138.90380859375 disk.dm-1.out‚ 453.87109375 disk.rbd1.out‚ 4.29296875 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 2.5950746e+07 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36138.90380859375 disk.sda3.out‚ 453.87109375 disk.drbd2265.in‚0¹282’Ç'‚(:‡ øæäÔ¼Ž‡~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.724' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.9448 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ñýÇsÈ'‚(:è øæäÔ€ ÷ network.sent‚ 2.6058888e+07$ disk./.usageGB‚260.9164047241211 disk.drbd2080.out‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚36191.53369140625$ memory_percent‚94.19772624969482 proc.cpu.threads‚17 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 454.64453125 disk.rbd1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 950.64453125 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.29296875 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375 disk.drbd2080.in‚0 disk.sda2.in‚0# disk.dm-0.out‚36191.53369140625 disk.sda1.out‚0& disk.nvme0n1.out‚36191.53369140625 network.recv‚ 3.119267e+06* disk./.usagePercent‚17.510318844481116 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 454.64453125 disk.sda2.out‚0 disk.sda3.out‚ 454.64453125* proc.memory.percent‚16.921377182006836 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 cpu‚25.33136748779024òÖÚì2É'- _wandbruntime‚8589  _runtime‚8589cJp&ÈÊ' & lr-AdamW/pg1‚5.791634230260218e-06 & lr-AdamW/pg2‚5.791634230260218e-06 trainer/global_step‚10799 % _timestamp‚1.7884250898107798e+09 _runtime‚8589.548312558 _step‚884¡µ·ÚÍË'Ç & lr-AdamW/pg1‚5.791634230260218e-06 & lr-AdamW/pg2‚5.791634230260218e-06 trainer/global_step‚10799 % _timestamp‚1.7884250898107798e+09 _runtime‚8589.548312558 _step‚884ônOq32Ì'- _wandbruntime‚8589  _runtime‚8589 W§š› Í'áO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143018368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9479596682752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚694417564511232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_stçÔ8 Lart/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9479764500992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143017923 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚636229565824512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788425089811486e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737388 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627192066559488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51519807 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10799 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842902528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156826622 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _runtime‚8589.549228306 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610768365257728 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91498116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038צì'Î'æO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91498118 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143018368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚684937991768576 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9479596682752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚694417564511232 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9479764500992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143017923 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚636229565824512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 $ _timestamp‚1.788425089811486e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 203737388 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚627192066559488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 360564010 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156826620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9003577221132 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚885 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51519807 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚684937967828480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚627192048556544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚610768348218368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚627192048556544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚636228891459072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚627192066559488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚10799 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚636228891459072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9036842902528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9037499265024 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156826622 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203737388 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 _runtime‚8589.549228306 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚610768365257728 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9037499265024 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 360543970 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚619772596916616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91498116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9004231658888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156826622 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156826620 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844õ•ßI~2Ï'- _wandbruntime‚8590  _runtime‚8590BÓ üò&Ð'ìM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91506284 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚627248401595392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360576440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 _step‚886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094060 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚627248419598336 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚451 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9480448524288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9037662330368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480067090432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684999642506240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91506286 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656096 _runtime‚8590.229411715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143031049 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203755868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚694480260932608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143030596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619827646005024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10799 % _timestamp‚1.7884250904912324e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738304512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚610823269388288 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156840638 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203735804 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9038318706176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9037662330368 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚636286063925760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚694480067090432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚684999642506240 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91506286 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9004393656096 _runtime‚8590.229411715 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156840636 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143031049 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚684999618566144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360596506 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360596506 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203755868 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚694480260932608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚627248401595392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143030596 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚619827646005024 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10799 % _timestamp‚1.7884250904912324e+09 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TɦL©v@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚636286738304512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚453 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚610823269388288 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156840638 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203755868 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚627248419598336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156840638 _step‚886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9005048094060öj÷º˜2Ò'- _wandbruntime‚8590  _runtime‚8590G¬÷vaÓ'Û %loss/total_loss‚49.29651424881167 0loss/SRqqt3/detection_loss‚1.699600749131706 % _timestamp‚1.7884250904932804e+09 0loss/FRt2/assignment_loss‚3.4466238799302475 /loss/FRt3/assignment_loss‚3.243694732773979 0loss/FRt4/assignment_loss‚1.6852977265391433 2loss/SRqqt1/assignment_loss‚1.5491041225749393 1loss/SRqqt2/assignment_loss‚2.085351334563617 1loss/FBt2/assignment_loss‚0.41514839418196914 /loss/FRt1/detection_loss‚2.8908811417119256 1loss/SRqqt1/detection_loss‚2.0453220470700155 /loss/FBt1/detection_loss‚2.3583749261769382 /loss/FBt4/detection_loss‚0.7730222617974505 /loss/FRt3/detection_loss‚2.3112071126980602 _runtime‚8590.230708542 2loss/FBt3/assignment_loss‚0.041765764355659485 1loss/FBt4/assignment_loss‚0.11138376832235321 _step‚887 epoch‚8 0loss/FRt1/assignment_loss‚3.4828482229134132 2loss/SRqqt4/assignment_loss‚0.8581964558123478 1loss/FBt1/assignment_loss‚0.26480427374603593 2loss/SRqqt3/assignment_loss‚1.1594274753911629 1loss/SRqqt2/detection_loss‚2.2836867745580345 /loss/FBt2/detection_loss‚2.9751387238502502 trainer/global_step‚10799 /loss/FRt2/detection_loss‚2.9970178085824717 /loss/FRt4/detection_loss‚1.4241048530920555 1loss/SRqqt4/detection_loss‚1.1127563291362355 .loss/FBt3/detection_loss‚8.081755369901657„ºÎLfÔ'à _step‚887 epoch‚8 0loss/FRt1/assignment_loss‚3.4828482229134132 2loss/SRqqt4/assignment_loss‚0.8581964558123478 1loss/FBt1/assignment_loss‚0.26480427374603593 2loss/SRqqt3/assignment_loss‚1.1594274753911629 1loss/SRqqt2/detection_loss‚2.2836867745580345 /loss/FBt2/detection_loss‚2.9751387238502502 trainer/global_step‚10799 /loss/FRt2/detection_loss‚2.9970178085824717 /loss/FRt4/detection_loss‚1.4241048530920555 1loss/SRqqt4/detection_loss‚1.1127563291362355 .loss/FBt3/detection_loss‚8.081755369901657 %loss/total_loss‚49.29651424881167 0loss/SRqqt3/detection_loss‚1.699600749131706 % _timestamp‚1.7884250904932804e+09 0loss/FRt2/assignment_loss‚3.4466238799302475 /loss/FRt3/assignment_loss‚3.243694732773979 0loss/FRt4/assignment_loss‚1.6852977265391433 2loss/SRqqt1/assignment_loss‚1.5491041225749393 1loss/SRqqt2/assignment_loss‚2.085351334563617 1loss/FBt2/assignment_loss‚0.41514839418196914 /loss/FRt1/detection_loss‚2.8908811417119256 1loss/SRqqt1/detection_loss‚2.0453220470700155 /loss/FBt1/detection_loss‚2.3583749261769382 /loss/FBt4/detection_loss‚0.7730222617974505 /loss/FRt3/detection_loss‚2.3112071126980602 _runtime‚8590.230708542 2loss/FBt3/assignment_loss‚0.041765764355659485 1loss/FBt4/assignment_loss‚0.11138376832235321÷nzΑÕ'‚(:† ‡çäÔ°âœ~ gpu.0.gpu‚7.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f‰ZêrÖ'‚(:ç ‡çäÔ§õ£€ disk.nvme0c0n1.in‚4.6875* disk./.usagePercent‚17.510629125612613 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0$ memory_percent‚94.19810771942139( proc.memory.availableMB‚ 950.58203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.6875& disk.nvme0n1.out‚36233.02783203125 disk.rbd0.in‚0 disk.sda3.out‚ 455.0546875 cpu‚25.330875790629204 disk.dm-1.out‚ 455.0546875 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.29296875 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 3.122822e+06# disk.dm-0.out‚36233.02783203125 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.sda.out‚ 455.0546875 network.sent‚ 2.606422e+07% disk./.usageGB‚260.92102813720703 disk.sda1.out‚0( disk.nvme0c0n1.out‚36233.02783203125 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda2.out‚0Zëú×'‚(:’ –çäÔáé’~ gpu.0.gpu‚53.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.504999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÃsxuØ'‚(:ê –çäÔ£òú€& disk.nvme0n1.out‚36291.50830078125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0# disk.dm-0.out‚36291.50830078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚36291.50830078125 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚260.92680740356445 disk.sda3.out‚ 455.94140625 disk.dm-1.in‚ 126.77734375 disk.drbd2266.out‚0$ memory_percent‚94.19808387756348' proc.memory.availableMB‚ 950.5859375" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 455.94140625 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 455.94140625 disk.dm-0.in‚4.6875 network.sent‚ 2.6173125e+07 disk.sda2.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.921377182006836 network.recv‚ 3.129513e+06* disk./.usagePercent‚17.511016977026987 disk.drbd2080.in‚0 disk.rbd1.out‚ 4.44140625 cpu‚25.330383857860898æeœ’Ù'‚(:‡ ¥çäÔ•Š®~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì™¾tÚ'‚(:é ¥çäÔ𗚀# disk.dm-0.out‚36335.29736328125 disk.rbd1.out‚ 4.44140625( proc.memory.availableMB‚ 950.58203125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚36335.29736328125 disk.sda3.out‚ 456.27734375 disk.drbd2080.in‚0 disk.sda1.out‚0$ disk./.usageGB‚260.9246292114258 disk.nvme0n1.in‚4.6875 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 456.27734375" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 cpu‚25.329931130536405 network.sent‚ 2.6179087e+07 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.133515e+06 disk.sda2.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚17.51087079672494 disk.drbd1716.in‚0& disk.nvme0n1.out‚36335.29736328125 disk.rbd0.out‚0 disk.dm-1.out‚ 456.27734375 disk.sda.in‚ 126.77734375$ memory_percent‚94.19810771942139ÀÌc]2Û'- _wandbruntime‚8626  _runtime‚8626[ºoÄÇÜ'Á _step‚888 & lr-AdamW/pg1‚5.356424669735467e-06 & lr-AdamW/pg2‚5.356424669735467e-06 trainer/global_step‚10849 $ _timestamp‚1.788425126431077e+09 _runtime‚8626.168602719I*æÌÝ'Æ trainer/global_step‚10849 $ _timestamp‚1.788425126431077e+09 _runtime‚8626.168602719 _step‚888 & lr-AdamW/pg1‚5.356424669735467e-06 & lr-AdamW/pg2‚5.356424669735467e-06ø\UÈ‚2Þ'- _wandbruntime‚8626  _runtime‚8626¨ >Mè'ß'âO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51748508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143655026 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 trainer/global_step‚10849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190862 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157527420 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91906518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688020504711680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204663440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204643402 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639087529207296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204663440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚688020528651776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622558638784024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884251264318206e+09 _runtime‚8626.169548324 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529207296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630009718501888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697543213275648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697543034100224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998296 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639088203572736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91906516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1„™ñ'$à'çO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9077828708352 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143654550 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9078485070848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157527420 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 362170822 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143655026 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚476 trainer/global_step‚10849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚889 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 362190862 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157527420 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚613513554746368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91906518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚688020504711680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630009718501888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630009700498944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204663440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204643402 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚639087529207296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204663440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚688020528651776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚622558638784024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚622557967306908 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884251264318206e+09 _runtime‚8626.169548324 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚639087529207296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚630009718501888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697543213275648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697543034100224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9045066998296 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚639088203572736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 TGDevi­$ü!ÆceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91906516 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51748508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157527422 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157541436 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701673984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304512000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788425127122189e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204661856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630066071540736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143667731 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143667258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51753045 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376052736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9523544632832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9079304512000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9045883433468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622613687872432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697605528082944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 204661856 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 362223358 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630066071540736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204681920 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚613568475916288 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143667731 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143667258 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 362203292 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51753045 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚639145376052736 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 362223358 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 362203292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91914684 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚688082155449344 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9523544632832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9079304512000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9045883433468 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204681920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630066071540736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚622613687872432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚697605528082944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157541436 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚639144701673984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9079304512000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204661856 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚473 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788425127122189e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944úã_”2ä'- _wandbruntime‚8626  _runtime‚8626o‚„`å'Ú 1loss/FBt4/assignment_loss‚0.12915000851040526 .loss/FRt3/detection_loss‚2.406689797286634 $ _timestamp‚1.788425127124359e+09 0loss/FBt2/assignment_loss‚0.3091198260979411 0loss/FRt4/assignment_loss‚1.5327936762669048 0loss/FBt1/assignment_loss‚0.6609625155106187 /loss/FBt3/detection_loss‚2.3656972646713257 /loss/FBt4/detection_loss‚0.5828817503915841 &loss/total_loss‚40.229273673471255 1loss/SRqqt2/detection_loss‚2.7497503338036715 1loss/SRqqt3/detection_loss‚1.3423360123084143 /loss/FBt2/detection_loss‚2.8060333999720486 .loss/FRt2/detection_loss‚3.117120643456777 2loss/SRqqt2/assignment_loss‚1.6277056490933453 /loss/FRt4/detection_loss‚1.7136657468547098 _runtime‚8626.861784249 2loss/SRqqt3/assignment_loss‚1.2446381051666462 1loss/SRqqt1/detection_loss‚1.6660264634660311 epoch‚8 0loss/FRt1/assignment_loss‚2.6528096337651097 2loss/SRqqt1/assignment_loss‚1.1531574161102374 /loss/FRt1/detection_loss‚2.2585237275722414 _step‚891 1loss/FBt3/assignment_loss‚0.32391467422712594 0loss/FRt2/assignment_loss‚2.9127043971308955 0loss/FRt3/assignment_loss‚2.7010700371530323 2loss/SRqqt4/assignment_loss‚0.9969846495737632 1loss/SRqqt4/detection_loss‚1.2092116442819436 /loss/FBt1/detection_loss‚1.7663263007998466 trainer/global_step‚10849z€™eæ'ß .loss/FRt3/detection_loss‚2.406689797286634 $ _timestamp‚1.788425127124359e+09 0loss/FBt2/assignment_loss‚0.3091198260979411 0loss/FRt4/assignment_loss‚1.5327936762669048 0loss/FBt1/assignment_loss‚0.6609625155106187 /loss/FBt3/detection_loss‚2.3656972646713257 /loss/FBt4/detection_loss‚0.5828817503915841 &loss/total_loss‚40.229273673471255 1loss/SRqqt2/detection_loss‚2.7497503338036715 1loss/SRqqt3/detection_loss‚1.3423360123084143 /loss/FBt2/detection_loss‚2.8060333999720486 .loss/FRt2/detection_loss‚3.117120643456777 2loss/SRqqt2/assignment_loss‚1.6277056490933453 /loss/FRt4/detection_loss‚1.7136657468547098 _runtime‚8626.861784249 2loss/SRqqt3/assignment_loss‚1.2446381051666462 1loss/SRqqt1/detection_loss‚1.6660264634660311 epoch‚8 0loss/FRt1/assignment_loss‚2.6528096337651097 2loss/SRqqt1/assignment_loss‚1.1531574161102374 /loss/FRt1/detection_loss‚2.2585237275722414 _step‚891 1loss/FBt3/assignment_loss‚0.32391467422712594 0loss/FRt2/assignment_loss‚2.9127043971308955 0loss/FRt3/assignment_loss‚2.7010700371530323 2loss/SRqqt4/assignment_loss‚0.9969846495737632 1loss/SRqqt4/detection_loss‚1.2092116442819436 /loss/FBt1/detection_loss‚1.7663263007998466 trainer/global_step‚10849 1loss/FBt4/assignment_loss‚0.12915000851040526ûe©NÀ’ç'‚(:‡ ´çäÔèñ¦ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ýuè'‚(:ê ´çäԹ럂 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.dm-1.out‚ 456.96484375 disk.rbd1.in‚0 network.sent‚ 2.6185395e+07 disk.drbd1716.out‚0& disk.nvme0n1.out‚36405.45166015625 disk.sda3.in‚ 126.77734375# disk.dm-0.out‚36405.45166015625 cpu‚25.32943418426065 disk.rbd0.out‚0 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1095.8046875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.31173896789551" proc.memory.rssMB‚ 2772.3984375 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚36405.45166015625 disk.sda3.out‚ 456.96484375 disk.drbd2080.out‚0* disk./.usagePercent‚17.512872519635163 disk.sda.out‚ 456.96484375* proc.memory.percent‚16.921377182006836$ disk./.usageGB‚260.9544563293457 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.drbd2266.in‚0 disk.rbd1.out‚ 4.4736328125 disk.dm-1.in‚ 126.77734375 network.recv‚ 3.13815e+06 disk.sda1.in‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0q²Ð’é'‚(:‡ ÃçäÔ…³Š~ gpu.0.gpu‚54.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.703' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÝÜuê'‚(:ê ÃçäÔ–ú disk.drbd1716.in‚0& disk.nvme0n1.out‚36445.13525390625# memory_percent‚93.3117151260376* disk./.usagePercent‚17.514177390037187 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.6293329e+07 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 457.43359375( disk.nvme0c0n1.out‚36445.13525390625* proc.memory.percent‚16.921377182006836$ disk./.usageGB‚260.9738998413086 disk.sda.out‚ 457.43359375 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 network.recv‚ 3.143556e+06# disk.dm-0.out‚36445.13525390625 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.4736328125 disk.drbd1716.out‚0 cpu‚25.33210427830455 disk.drbd2080.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 457.43359375 disk.drbd2265.in‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1095.80859375 disk.drbd2080.in‚0 disk.sda1.in‚0Ñaµœë'‚(:‘ ÒçäÔ²œû} gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.218' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.843600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ ærì'‚(:ç ÒçäÔ´û€ disk.nvme0c0n1.in‚4.6875" proc.memory.rssMB‚ 2772.3984375 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 network.sent‚ 2.6299352e+07 disk.drbd2080.in‚0" disk.dm-0.out‚36486.3974609375 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0 disk.drbd2265.in‚0$ memory_percent‚93.31169128417969* proc.memory.percent‚16.921377182006836 disk.rbd1.in‚0 network.recv‚ 3.148171e+06* disk./.usagePercent‚17.513328725061182 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 cpu‚25.34049484917453 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚36486.3974609375 disk.sda3.out‚ 458.10546875 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.53662109375% disk./.usageGB‚260.96125411987305 disk.dm-1.out‚ 458.10546875 disk.sda.out‚ 458.10546875% proc.memory.availableMB‚ 1095.8125 disk.dm-0.in‚4.6875 disk.sda2.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚36486.3974609375 proc.cpu.threads‚17 disk.nvme0n1.in‚4.6875Äã$’í'‚(:‡ áçäÔ»ñ« gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚121.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.3866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Öš©iî'‚(:Þ áçäÔǤ€ disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚458.4375$ memory_percent‚93.31173896789551( proc.memory.availableMB‚ 1095.8046875 cpu‚25.349212716671452* disk./.usagePercent‚17.515359888837843' disk.nvme0c0n1.out‚36516.0810546875 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚458.4375 disk.sda.out‚458.4375* proc.memory.percent‚16.921377182006836 disk.nvme0n1.in‚4.6875 disk.sda.in‚ 126.77734375 network.sent‚ 2.6305175e+07 network.recv‚ 3.152476e+06$ disk./.usageGB‚260.9915199279785 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 4.53662109375 proc.cpu.threads‚17 disk.sda2.in‚0" disk.dm-0.out‚36516.0810546875 disk.rbd1.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2772.3984375% disk.nvme0n1.out‚36516.0810546875 disk.rbd0.in‚0AYÐ2ï'- _wandbruntime‚8697  _runtime‚8697ë‘õôj ð'ÈT ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚701417576855040 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9568204603904 _step‚892 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 158505428 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9122618917376 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚52116653 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795240448 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9122618917376 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339635712 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚633617687156224 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚8697.182619054 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 205596530 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 144603243 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚642740306073600 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚633617669153280 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚92486590 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 132992512 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 364117935 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 144603089 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52116501 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 364101956 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚642740306073600 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9122126087168 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101956 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 364117935 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚633617669153280 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚626088929947188 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 158505426 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚616999881139200 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚642739795240448 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚616999898178560 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9089031768628H LS*ñ'ÍT VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 205596530 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 144603243 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚414 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚642740306073600 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚633617669153280 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚92486590 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 132992512 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 861853521 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚430 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 364117935 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 144603089 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚52116501 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚414 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 364101956 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 9643699025 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚642740306073600 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9122126087168 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 364101956 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 364117935 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚633617669153280 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 490914392 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚626088929947188 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 158505426 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 868220928 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚30 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚616999881139200 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚642739795240448 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚616999898178560 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9089031768628 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚844 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚152 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚701417576855040 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9568204603904 _step‚892 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 158505428 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9122618917376 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 158505428 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚52116653 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚642739795240448 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9122618917376 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339635712 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚633617687156224 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 _runtime‚8697.182619054 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚430üHŠÚ2ò'- _wandbruntime‚8698  _runtime‚8698kfžõY)ó'ÓR QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚889 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚617035174898688 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9122491213824 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 205603832 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9122491213824 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚893 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 158516234 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚92493092 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 9568678782976 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 205619809 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 205603832 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 158516232 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚844 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9088904463644 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚844 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 158516234 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 158516232 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚10880 _runtime‚8698.082194882 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚633654517185024 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 132992512 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9089395378036 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 510833152 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701455817715200 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚642776990395904 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚691887114992128 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501229056 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚642776990395904 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚52121029 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚701455684722688 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 205619809 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339635712 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120064 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 144614121 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚701455817715200 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚642776990395904 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 510833152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 9326034944 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚691887114992128 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚642777501229056 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚642776990395904 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚633654517185024 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 492830208 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚52121029 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚701455684722688 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚430 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 205619809 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚626124079362332 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 339635712 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 868220928 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 364120064 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 861853521 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 863815680 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 144614121 B&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.092741958796978 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5222154855728149 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856947660446167 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10919539630413055 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917669892311096 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8127003908157349 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19058576226234436 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4687500298023224 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.7105262875556946 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13281191885471344 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4180329144001007 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2651928663253784 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08859649300575256 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4645102918148041 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.054178666323423386 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5740189552307129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12747064232826233 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01904761977493763 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4674557149410248 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08115965127944946 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6392342448234558 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20226655900478363 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚894 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5900321006774902 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 jet/accuracy_4_of_7‚0 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 epoch‚8 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9485005736351013 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20500053465366364 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26027020812034607 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.58122718334198 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6067915558815002 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987658202648163 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36118385195732117 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6291666030883789 +jet/accuracy_4_of_4‚0.00779682444408536 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16362716257572174 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105388641357 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5346416234970093 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884252107366068e+09 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5763912796974182 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982101321220398 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3803599178791046 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6382524371147156 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06708301603794098 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_5‚0.010906298644840717 particle/accuracy_4_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5905314683914185 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13737839460372925 particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28981485962867737 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12444445490837097 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16421084105968475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29796841740608215 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6051732301712036 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_2_of_4‚0.38201192021369934 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.59333336353302 jet/accuracy_6_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5150984525680542 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35158059000968933 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5798548460006714 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30367687344551086 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883156418800354 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5772370100021362 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03420886769890785 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904561042785645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1770610511302948 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1146494671702385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5776606202125549 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47057053446769714 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020120136439800262 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4115327000617981 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5674785375595093 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5071062445640564 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09184826165437698 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5733906626701355 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5142719745635986 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5497504472732544 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12345939129590988 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5694556832313538 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6013177037239075 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12283584475517273 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1368415802717209 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_4‚0.9350476264953613 particle/accuracy_3_of_4‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029394814744591713 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2401861995458603 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6971045732498169 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09560699760913849 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5563435554504395 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04127565398812294 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09312482178211212 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22862105071544647 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43313953280448914 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2905607521533966 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0355827659368515 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16238261759281158 *jet/accuracy_1_of_5‚0.9596901535987854 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2608390748500824 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17120522260665894 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.206686332821846 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_4‚1 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1550290435552597 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 particle/accuracy_2_of_4‚1 +jet/accuracy_5_of_6‚0.01923076994717121 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.55753493309021 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3138881325721741 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5801607966423035 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12056566029787064 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5153863430023193 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01655055396258831 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 _runtime‚8710.476084422 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 +jet/accuracy_2_of_2‚0.11059599369764328 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9827637076377869 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15701927244663239 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.47374728322029114 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4628641903400421 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5539920926094055 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt4SRqqt4FBt/ev(åc3ä6ent_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912527799606323 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09544505178928375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5587928295135498 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08654573559761047 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8941898941993713 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07450052350759506 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 +jet/accuracy_2_of_3‚0.25286760926246643 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6021825671195984 particle/accuracy_2_of_3‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 *jet/accuracy_2_of_5‚0.4963645935058594 6validation_average_jet_accuracy‚0.4579344689846039 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_7_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13383840024471283 (particle/f_score‚0.39049333333969116 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5692492723464966 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925991892814636 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6086691617965698 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17099179327487946 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05842912197113037 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01833970658481121 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5247039198875427 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 *loss/val_total_loss‚42.504313744207074 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5369551777839661 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5189749598503113 +jet/accuracy_4_of_6‚0.01923076994717121 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2390872836112976 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.6089743971824646 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5720941424369812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.02777777798473835 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_2‚0.7859147191047668 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5057238936424255 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 jet/accuracy_7_of_7‚0 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.59258633852005 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36679893732070923 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5094828009605408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908512830734253 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09245384484529495 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26350995898246765 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5922349691390991 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5798548460006714 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9902138113975525 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5051313638687134 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5500504970550537 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6269605159759521 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18802188336849213 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1258000135421753 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012683739885687828 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16421084105968475 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37872204184532166 particle/accuracy_2_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1378318965435028 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5634743571281433 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.026888342574238777 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855872988700867 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12665802240371704 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3103208839893341 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33563289046287537 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008046514354646206 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5710253119468689 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.502875030040741 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031256950460374355 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896060228347778 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5540046095848083 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.272632360458374 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979490041732788 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1945713758468628 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17017951607704163 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5413932204246521 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_7‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2461550384759903 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5616080164909363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4594298303127289 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4718954861164093 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5810914039611816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41324108839035034 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12934249639511108 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1003294363617897 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018490785732865334 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_4‚0.06900130957365036 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0‰y Iù'Ѧ 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2905607521533966 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0355827659368515 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16238261759281158 *jet/accuracy_1_of_5‚0.9596901535987854 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2608390748500824 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17120522260665894 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.206686332821846 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_3‚1 particle/accuracy_4_of_4‚1 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.34375 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1550290435552597 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5833333730697632 particle/accuracy_2_of_4‚1 +jet/accuracy_5_of_6‚0.01923076994717121 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.55753493309021 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3138881325721741 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5801607966423035 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12056566029787064 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5153863430023193 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01655055396258831 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 _runtime‚8710.476084422 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 +jet/accuracy_2_of_2‚0.11059599369764328 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9827637076377869 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15701927244663239 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.47374728322029114 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4628641903400421 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5539920926094055 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9912527799606323 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09544505178928375 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5587928295135498 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08654573559761047 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8941898941993713 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07450052350759506 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12056738138198853 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 +jet/accuracy_2_of_3‚0.25286760926246643 particle/accuracy_3_of_3‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6021825671195984 particle/accuracy_2_of_3‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 *jet/accuracy_2_of_5‚0.4963645935058594 6validation_average_jet_accuracy‚0.4579344689846039 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_7_of_7‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13383840024471283 (particle/f_score‚0.39049333333969116 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5692492723464966 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925991892814636 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6086691617965698 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17099179327487946 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.05842912197113037 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.01833970658481121 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5247039198875427 jet/accuracy_5_of_7‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.17905405163764954 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 *loss/val_total_loss‚42.504313744207074 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5369551777839661 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.5189749598503113 +jet/accuracy_4_of_6‚0.01923076994717121 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2390872836112976 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_6‚0.6089743971824646 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5720941424369812 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.02777777798473835 particle/accuracy_2_of_7‚1 particle/accuracy_3_of_7‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_1_of_2‚0.7859147191047668 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.5057238936424255 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 jet/accuracy_7_of_7‚0 7"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.59258633852005 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.36679893732070923 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5094828009605408 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9908512830734253 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09245384484529495 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26350995898246765 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5922349691390991 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5798548460006714 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9902138113975525 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5051313638687134 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5500504970550537 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6269605159759521 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18802188336849213 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1258000135421753 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012683739885687828 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16421084105968475 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37872204184532166 particle/accuracy_2_of_5‚1 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35092589259147644 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1378318965435028 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5634743571281433 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5568452477455139 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.026888342574238777 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855872988700867 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12665802240371704 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3103208839893341 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33563289046287537 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008046514354646206 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5710253119468689 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.502875030040741 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031256950460374355 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896060228347778 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5540046095848083 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.272632360458374 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979490041732788 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1945713758468628 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.17017951607704163 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5413932204246521 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_7‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2461550384759903 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5616080164909363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4594298303127289 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4718954861164093 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5810914039611816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41324108839035034 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12934249639511108 ,&Purity/2FRt4SRqqt1FBtÃ8Ý¿JJ/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1003294363617897 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.018490785732865334 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_4‚0.06900130957365036 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41073012351989746 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913349747657776 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5157894492149353 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.35282737016677856 jet/accuracy_1_of_7‚1 trainer/global_step‚10880 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.509300172328949 (particle/accuracy‚0.8488396406173706 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.33541664481163025 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.6342592835426331 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6666666865348816 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6256409883499146 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.5754877924919128 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6666666865348816 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.092741958796978 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5222154855728149 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856947660446167 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.10919539630413055 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917669892311096 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8127003908157349 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19058576226234436 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4687500298023224 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.7105262875556946 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13281191885471344 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4180329144001007 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2651928663253784 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.08859649300575256 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4645102918148041 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.054178666323423386 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43827158212661743 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5740189552307129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12747064232826233 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01904761977493763 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4674557149410248 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08115965127944946 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6392342448234558 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20226655900478363 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚894 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5900321006774902 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 jet/accuracy_4_of_7‚0 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 epoch‚8 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9485005736351013 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20500053465366364 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26027020812034607 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.58122718334198 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6067915558815002 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.20987658202648163 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36118385195732117 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6291666030883789 +jet/accuracy_4_of_4‚0.00779682444408536 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16362716257572174 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105388641357 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.1875 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5346416234970093 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884252107366068e+09 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5763912796974182 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982101321220398 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3803599178791046 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6382524371147156 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.06708301603794098 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_5‚0.010906298644840717 particle/accuracy_4_of_6‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5905314683914185 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13737839460372925 particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28981485962867737 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12444445490837097 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16421084105968475 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29796841740608215 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6051732301712036 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 +jet/accuracy_2_of_4‚0.38201192021369934 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.59333336353302 jet/accuracy_6_of_7‚0 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5150984525680542 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35158059000968933 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5798548460006714 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30367687344551086 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883156418800354 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5772370100021362 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03420886769890785 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9904561042785645 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1770610511302948 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1146494671702385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5776606202125549 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47057053446769714 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020120136439800262 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4115327000617981 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5674785375595093 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5071062445640564 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 particle/accuracy_3_of_5‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09184826165437698 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5733906626701355 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_6_of_7‚1 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.5142719745635986 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5497504472732544 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12345939129590988 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5694556832313538 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6013177037239075 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12283584475517273 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1368415802717209 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_1_of_4‚0.9350476264953613 particle/accuracy_3_of_4‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.029394814744591713 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2401861995458603 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6971045732498169 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09560699760913849 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5563435554504395 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04127565398812294 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09312482178211212 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22862105071544647 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.43313953280448914 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1þ\Úþ‘ú'‚(:† ÿçäÔÀ‘ ~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.294' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4588 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì?¾†tû'‚(:é ÿçäÔ඀) disk./.usagePercent‚17.51765704444258 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚ 3.163318e+06 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1425.41015625" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.6419713e+07# disk.dm-0.out‚37102.68798828125 disk.sda.out‚ 459.6484375 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.921377182006836% disk./.usageGB‚261.02574920654297 disk.dm-0.in‚4.6875 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.599609375 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚37102.68798828125 disk.sda3.out‚ 459.6484375 cpu‚25.361997249601263 disk.drbd2080.in‚0 disk.dm-1.out‚ 459.6484375 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚37102.68798828125 proc.cpu.threads‚17$ memory_percent‚91.29998683929443Û­E2ü'- _wandbruntime‚8728  _runtime‚8728º% Èý' & lr-AdamW/pg1‚4.937675856347752e-06 & lr-AdamW/pg2‚4.937675856347752e-06 trainer/global_step‚10899 % _timestamp‚1.7884252283687623e+09 _runtime‚8728.106318646 _step‚895p¯aÍþ'Ç _step‚895 & lr-AdamW/pg1‚4.937675856347752e-06 & lr-AdamW/pg2‚4.937675856347752e-06 trainer/global_step‚10899 % _timestamp‚1.7884252283687623e+09 _runtime‚8728.106318646ÿc¬42ÿ'- _wandbruntime‚8728  _runtime‚8728£85Üè'€(âO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644290153105408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589070158848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 trainer/global_step‚10899 % _timestamp‚1.7884252283694658e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158909000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206031417 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9141834484224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693489236745728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9141834484224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158908998 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9108172400030 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635147662258688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9108826837786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078127729152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206051455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206051455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9141834484224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚693489236745728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9141834484224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158908998 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 91081724000 9¨\"30 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚635147662258688 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9108826837786 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚644289478739968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92724668 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚703078127729152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚635147662258688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚627591111999258 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚618482285161472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206051455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚635147644255744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚618482268122112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206051455 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚435 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚644290153105408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364940415 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚693489212805632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9589070158848 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206031417 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚635147644255744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 trainer/global_step‚10899 % _timestamp‚1.7884252283694658e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158909000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206031417 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144997252 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703140815947264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚8728.792585644 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52264841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚644347325585408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9143310287872 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9589928463872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9589758561792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069935 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158923016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144997679 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653912064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚627646832564990 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788425229054428e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693550863543296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9108988834994 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9589928463872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9589758561792 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206069935 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9109643272958 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92732836 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158923016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚10899 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364992951 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144997679 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚427 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚693550887483392 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚635204015297536 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9142653912064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364972885 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚618537172252672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚644346651206656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9142653912064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158923016 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚627646832564990 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚897 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 193842176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 $ _timestamp‚1.788425229054428e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚693550863543296 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144997252 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚703140815947264 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚8728.792585644^f$2…(- _wandbruntime‚8728  _runtime‚8728–Ç<ÿ]†(× 0loss/FRt4/assignment_loss‚1.2461889355347073 1loss/FBt1/assignment_loss‚0.39246380393160507 0loss/FBt3/assignment_loss‚0.1072781990154894 1loss/SRqqt2/detection_loss‚2.3834748813291875 /loss/FBt1/detection_loss‚2.3444896526634693 2loss/SRqqt4/assignment_loss‚0.7971159977305968 1loss/FBt2/assignment_loss‚0.33930617960036863 1loss/FBt4/assignment_loss‚0.04109099228281268 .loss/FBt3/detection_loss‚2.934003305435181 _step‚898 /loss/FRt3/detection_loss‚2.0744643233039164 0loss/SRqqt3/detection_loss‚1.562260483028525 1loss/SRqqt4/detection_loss‚0.9174746277139467 2loss/SRqqt2/assignment_loss‚1.4416935701922673 /loss/FRt4/detection_loss‚1.3779385878286998 trainer/global_step‚10899 2loss/SRqqt3/assignment_loss‚1.3325904769412542 _runtime‚ 8728.7935905 0loss/FRt2/assignment_loss‚3.4035437141145977 /loss/FRt3/assignment_loss‚2.441159712184559 /loss/FRt2/detection_loss‚2.7858705145972116 .loss/FBt2/detection_loss‚2.374124105487551 /loss/FRt1/assignment_loss‚3.571914631873369 /loss/FRt1/detection_loss‚1.9834134876728118 /loss/FBt4/detection_loss‚0.7111914529976073 &loss/total_loss‚39.896335874883135 2loss/SRqqt1/assignment_loss‚1.4255517717213073 1loss/SRqqt1/detection_loss‚1.9077324677020946 epoch‚9 % _timestamp‚1.7884252290561707e+09ÿÜIœb‡(Ü 0loss/FRt4/assignment_loss‚1.2461889355347073 1loss/FBt1/assignment_loss‚0.39246380393160507 0loss/FBt3/assignment_loss‚0.1072781990154894 1loss/SRqqt2/detection_loss‚2.3834748813291875 /loss/FBt1/detection_loss‚2.3444896526634693 2loss/SRqqt4/assignment_loss‚0.7971159977305968 1loss/FBt2/assignment_loss‚0.33930617960036863 1loss/FBt4/assignment_loss‚0.04109099228281268 .loss/FBt3/detection_loss‚2.934003305435181 _step‚898 /loss/FRt3/detection_loss‚2.0744643233039164 0loss/SRqqt3/detection_loss‚1.562260483028525 1loss/SRqqt4/detection_loss‚0.9174746277139467 2loss/SRqqt2/assignment_loss‚1.4416935701922673 /loss/FRt4/detection_loss‚1.3779385878286998 trainer/global_step‚10899 2loss/SRqqt3/assignment_loss‚1.3325904769412542 _runtime‚ 8728.7935905 0loss/FRt2/assignment_loss‚3.4035437141145977 /loss/FRt3/assignment_loss‚2.441159712184559 /loss/FRt2/detection_loss‚2.7858705145972116 .loss/FBt2/detection_loss‚2.374124105487551 /loss/FRt1/assignment_loss‚3.571914631873369 /loss/FRt1/detection_loss‚1.9834134876728118 /loss/FBt4/detection_loss‚0.7111914529976073 &loss/total_loss‚39.896335874883135 2loss/SRqqt1/assignment_loss‚1.4255517717213073 1loss/SRqqt1/detection_loss‚1.9077324677020946 epoch‚9 % _timestamp‚1.7884252290561707e+09‚± Lˈ(‚(:’ ŽèäÔžãŒ~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚172.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚34.595800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eæ s¾‰(‚(:æ ŽèäÔ¥èõ€* proc.memory.percent‚16.921377182006836 disk.rbd0.out‚0" disk.dm-0.out‚37164.7783203125 network.recv‚ 3.169957e+06 disk.nvme0c0n1.in‚4.6875' disk.nvme0c0n1.out‚37164.7783203125 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-1.in‚ 126.77734375 proc.cpu.threads‚17) disk./.usagePercent‚17.51782063325696$ disk./.usageGB‚261.0281867980957 disk.dm-1.out‚ 460.30859375 disk.drbd2266.in‚0 cpu‚25.361459874113425 network.sent‚ 2.6š|@³544037e+07 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 460.30859375% disk.nvme0n1.out‚37164.7783203125 disk.sda3.out‚ 460.30859375 disk.drbd2080.out‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1425.3828125 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875 disk.drbd2265.in‚0# memory_percent‚91.3001537322998" proc.memory.rssMB‚ 2772.3984375ýŠ(‚(:… èäÔαŠ~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚171.77' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ß5n‹(‚(:ã èäÔ‹ºô€ disk.nvme0c0n1.in‚4.6875$ disk./.usageGB‚261.0453300476074 disk.drbd2265.out‚0 disk.sda.out‚ 460.69140625 disk.drbd1716.in‚0$ disk.nvme0n1.out‚37694.251953125 disk.sda2.out‚0 disk.drbd2265.in‚0! disk.dm-0.out‚37694.251953125 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1425.34765625" proc.memory.rssMB‚ 2772.3984375 network.recv‚ 3.175805e+06 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0" memory_percent‚91.300368309021 disk.sda1.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 460.69140625 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 460.69140625 disk.sda2.in‚0 network.sent‚ 2.6652283e+07 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375 disk.sda.in‚ 126.77734375* disk./.usagePercent‚17.518971131115833 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚37694.251953125* proc.memory.percent‚16.921377182006836 cpu‚25.36095577726908 disk.sda1.out‚0¶Œ¹’Œ(‚(:‡ ¬èäÔ„¼²~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Ä™'o(‚(:ä ¬èäÔ©ãñ( disk./.usagePercent‚17.5186516337131 disk.dm-1.out‚ 461.2578125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1424.8515625 disk.nvme0n1.in‚4.6875$ disk./.usageGB‚261.0405693054199 disk.drbd2080.in‚0# disk.dm-0.out‚37755.41259765625 disk.dm-1.in‚ 126.77734375 disk.drbd1716.in‚0( disk.nvme0c0n1.out‚37755.41259765625 network.sent‚ 2.665863e+07& disk.nvme0n1.out‚37755.41259765625 disk.sda2.out‚0 disk.sda3.out‚ 461.2578125 cpu‚25.360390443620265 disk.sda1.in‚0 network.recv‚ 3.18049e+06 disk.drbd2265.in‚0 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 461.2578125$ memory_percent‚91.30339622497559* proc.memory.percent‚16.921377182006836 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.dm-0.in‚4.6875 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0›Ç/Å2Ž(- _wandbruntime‚8765  _runtime‚8765›{UÅ(¿ & lr-AdamW/pg2‚4.535475056722148e-06 trainer/global_step‚10949 # _timestamp‚1.78842526536201e+09 _runtime‚ 8765.09954274 _step‚899 & lr-AdamW/pg1‚4.535475056722148e-06o,ìÊ(Ä & lr-AdamW/pg1‚4.535475056722148e-06 & lr-AdamW/pg2‚4.535475056722148e-06 trainer/global_step‚10949 # _timestamp‚1.78842526536201e+09 _runtime‚ 8765.09954274 _step‚899ƒ·æ2‘(- _wandbruntime‚8765  _runtime‚8765õ§Šç'’(áO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52484966 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145618036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647148790853632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206977507 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866666 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159609800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706203744110592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚696571773628928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9631970481664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438 trainer/global_step‚10949 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9631811052032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706203560740864 $ _timestamp‚1.788425265362712e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145617598 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206957469 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0õéšâì'“(æO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 145618036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93133068 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚621227491689472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366567267 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9183476652544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366587307 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚647148116488192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9183476652544 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚647148790853632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206977507 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚630377153866666 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 159609798 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 159609800 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚706203744110592 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚696571773628928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637965314201088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚647148790853632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9631970481664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637965314201088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637965296198144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206957469 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9149007739438 trainer/global_step‚10949 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚637965296198144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚647148116488192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9149662177194 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9182820290048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚696571749688832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9631811052032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚621227474650112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚630376482389550 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 _step‚900 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚706203560740864 $ _timestamp‚1.788425265362712e+09 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 145617598 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 159609798 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206957469 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52484966 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944„Ÿ.¶92”(- _wandbruntime‚8765  _runtime‚8765ççu+û•(ìM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647205963333632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145630656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚638021667239936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 trainer/global_step‚10949 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296093696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647205963333632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612366 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975923 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145630210 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52489418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚638021649236992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚638021649236992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52488974 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872rµ¥‘÷&–(ñM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9632832882688 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159623816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159623816 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159623814 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚621282395820032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚638021667239936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9183639717888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366599737 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚647205963333632 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145630656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚638021667239936 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚446 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚696633424366592 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9149824174402 trainer/global_step‚10949 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9184296093696 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647205963333632 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9150478612366 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 206975923 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚444 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 366619803 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647205288954880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145630210 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚630432874432398 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52489418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚638021649236992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366619803 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚638021649236992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93141236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52488974 BßîOc™(Ý 0loss/FRt2/assignment_loss‚2.9732714976583208 2loss/SRqqt1/assignment_loss‚1.4669802104864194 .loss/FRt2/detection_loss‚2.804292457444327 1loss/SRqqt1/detection_loss‚1.8630231865206544 1loss/SRqqt3/detection_loss‚1.9425089759752154 0loss/FRt3/assignment_loss‚2.5701542718465937 2loss/SRqqt2/assignment_loss‚1.5661467537283897 2loss/SRqqt3/assignment_loss‚1.3292912255662184 /loss/FBt4/detection_loss‚0.5786829074150978 _step‚902 2loss/FBt3/assignment_loss‚0.016197513323277235 0loss/SRqqt2/detection_loss‚2.551075649261475 0loss/SRqqt4/detection_loss‚1.030814494335038 _runtime‚8765.819272249 2loss/SRqqt4/assignment_loss‚0.9118353012839661 1loss/FBt2/assignment_loss‚0.15603794782280353 /loss/FRt1/detection_loss‚2.2693440929437294 epoch‚9 trainer/global_step‚10949 $ _timestamp‚1.788425266081847e+09 0loss/FRt1/assignment_loss‚2.9235499852742906 .loss/FRt3/detection_loss‚2.506466248700785 /loss/FBt1/detection_loss‚2.2926723957061768 1loss/FBt4/assignment_loss‚0.20994474514172645 %loss/total_loss‚40.90349641771701 1loss/FBt1/assignment_loss‚0.23414185905962118 /loss/FRt4/detection_loss‚1.8088614369875617 .loss/FBt2/detection_loss‚2.344773330471732 0loss/FRt4/assignment_loss‚1.6225474530032702 .loss/FBt3/detection_loss‚2.930882477760315†¢• €’š(‚(:‡ »èäԴ;~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚174.136' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.8272 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03>3:w›(‚(:ì »èäÔöž¢ disk.rbd1.out‚ 4.599609375& disk.nvme0n1.out‚37804.58056640625 disk.rbd0.out‚0 cpu‚25.359836885384528 disk.sda2.in‚0 disk.sda3.out‚ 461.66796875 disk.dm-1.out‚ 461.66796875* disk./.usagePercent‚17.520276257525396 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 461.66796875* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17$ disk./.usageGB‚261.0647773742676 disk.drbd2080.in‚0# disk.dm-0.out‚37804.58056640625 disk.rbd1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚37804.58056640625 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0) proc.memory.availableMB‚ 1424.56640625" proc.memory.rssMB‚ 2772.3984375 disk.nvme0c0n1.in‚4.6875 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875$ memory_percent‚91.30513668060303 network.sent‚ 2.6663898e+07 network.recv‚ 3.184131e+06 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375ØðËœ(‚(:’ ÊèäÔ¾¿©€ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.749' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚33.74979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N'/n(‚(:ã ÊèäÔ«ªô‚% disk./.usageGB‚261.09239959716797 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.out‚0 disk.dm-0.in‚4.6875# disk.dm-0.out‚37866.72705078125 disk.rbd1.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚37866.72705078125 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.52213000808083 disk.drbd2265.in‚0 disk.nvme0n1.in‚4.6875# proc.memory.availableMB‚950.375 cpu‚25.359299840009008 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 462.3828125 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0$ memory_percent‚94.19937133789062* proc.memory.percent‚16.921377182006836 disk.sda3.out‚ 462.3828125 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.599609375 disk.drbd1716.in‚0& disk.nvme0n1.out‚37866.72705078125" proc.memory.rssMB‚ 2772.3984375 disk.dm-1.out‚ 462.3828125 disk.nvme0c0n1.in‚4.6875 network.sent‚ 2.6772467e+07 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 network.recv‚ 3.190435e+06ëÕU2ž(- _wandbruntime‚8802  _runtime‚8802nÐÛ)ÈŸ( % _timestamp‚1.7884253023261244e+09 _runtime‚8802.063645534 _step‚903 & lr-AdamW/pg1‚4.149906088902472e-06 & lr-AdamW/pg2‚4.149906088902472e-06 trainer/global_step‚10999Žœy~Í (Ç % _timestamp‚1.7884253023261244e+09 _runtime‚8802.063645534 _step‚903 & lr-AdamW/pg1‚4.149906088902472e-06 & lr-AdamW/pg2‚4.149906088902472e-06 trainer/global_step‚10999‡È-œ2¡(- _wandbruntime‚8802  _runtime‚8802³¨îç'¢(áO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9674724317184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329211036160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52710998 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52710536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9674900524032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚709329010889216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516602 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146252004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640782948140544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368214159 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462458368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146252468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754236416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 IÓ'£(æO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207883521 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146252468 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93541468 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚650006754236416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 207903559 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚640782948140544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9674724317184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚633162524256958 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623972698217472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚462 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 207903559 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709329211036160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9223806095872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52710998 T±“Gæ*GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52710536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 368194119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9674900524032 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9189843078846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650007428601856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 160310600 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚709329010889216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9190497516602 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160310598 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146252004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93541470 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚633163195734074 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207883521 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699654286572032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650006754236416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚640782948140544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 368214159 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160310598 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176206848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9224462458368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872ˆT‚†C2¤(- _wandbruntime‚8802  _runtime‚8802Ì(Qò&¥(ìM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8802.754139134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52715439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52714992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624027585308672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93549636 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884253030160863e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368226589 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚709391714514432 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9225281899520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146265077 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9675753264640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9675593848320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207901975 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146264628 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926703104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160324616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513810 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922039 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640839301179392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚10999 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚633218916299806 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633218244822482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872ü¯Z÷&¦(ñM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224625523712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224625523712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 368246655 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 159416320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146265077 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚699715937309696 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9675753264640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9675593848320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚905 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9225281899520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207901975 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207901975 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146264628 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚650063926703104 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 160324616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9190659513810 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93549638 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 160324614 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207922039 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚640839301179392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 160324614 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚640839319182336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 trainer/global_step‚10999 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚633218916299806 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650064601081856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚640839301179392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚449 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚633218244822482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 368246655 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚8802.754139134 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚640839319182336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52715439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52714992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚624027585308672 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93549636 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884253030160863e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 368226589 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207922039 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚709391714514432 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9225281899520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 368226589 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208809573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652866066350080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52938124 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391984640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702736823455232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208829611 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643600618085888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚908 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712454624464896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626717904745472 % _timestamp‚1.7884253388475976e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626717887706112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9717617639936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52937686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9717777069568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448264192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9264791901696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643600618085888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643600600082944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265448264192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146887994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430ÿá0Bí'µ(çO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93949868 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚652866066350080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208809573 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚652866066350080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52938124 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚652865391984640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚93949870 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702736823455232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635948566124366 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208829611 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9231332856010 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚643600618085888 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161011398 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚908 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚712454624464896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚626717904745472 % _timestamp‚1.7884253388475976e+09 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚626717887706112 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161011398 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚652865391984640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 369841011 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208809573 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9717617639936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9230678418254 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9264791901696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702736847395328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369820971 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161011400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚438 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52937686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9717777069568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚643600600082944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265448264192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9264791901696 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚643600618085888 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208829611 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚643600600082944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265448264192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146887994 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128Œž•kë2¶(- _wandbruntime‚8839  _runtime‚8839¶ùâ"ò&·(ìM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146900120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828027 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564451328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853218 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828027 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9718461267968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚626772808876032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52942521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167214 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚11049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93958038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9718633267200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238830080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161025416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8839.253196602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507PÚŠ¡F¸(ñM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9718461267968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚626772808876032 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚626772791836672 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚643656953121792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9265611329536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52942521 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚636004958167214 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 trainer/global_step‚11049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RDDe«ÎTw±"viceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 161025416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚93958038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702798498132992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9718633267200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9232149291182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚652923238830080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702798474192896 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208848091 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 171999232 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369873507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 195939328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚643656971124736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 161025416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9265611329536 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369853441 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚8839.253196602 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369873507 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚652923238830080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146900120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚636004286689890 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208828027 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚652922564451328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9231494853218 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208828027 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚643656971124736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161025414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712{9æP2¹(- _wandbruntime‚8839  _runtime‚8839nÕ¿ò^º(Ø 1loss/SRqqt3/detection_loss‚1.7565697993169416 /loss/FBt1/detection_loss‚2.6877306832207575 trainer/global_step‚11049 _step‚910 0loss/FRt3/assignment_loss‚2.1208296418190002 epoch‚9 1loss/FBt3/assignment_loss‚0.06564388733305451 .loss/FRt1/detection_loss‚2.290331661023877 .loss/FBt3/detection_loss‚2.493212044239044 % _timestamp‚1.7884253395167952e+09 1loss/FBt4/assignment_loss‚0.12062941262459664 .loss/FRt2/detection_loss‚2.979726036389669 /loss/FRt3/detection_loss‚2.4206886386329476 0loss/FRt2/assignment_loss‚3.3760253768978696 0loss/FRt4/assignment_loss‚1.4773745328187942 1loss/SRqqt1/detection_loss‚1.8598497742985158 /loss/FBt2/detection_loss‚2.5914827035023618 /loss/FRt4/detection_loss‚1.6666184117396672 1loss/SRqqt4/detection_loss‚1.0273831586425122 _runtime‚8839.254262463 1loss/SRqqt2/assignment_loss‚1.800287685432547 2loss/SRqqt3/assignment_loss‚1.4643788637839399 1loss/FBt1/assignment_loss‚0.18458679349472126 %loss/total_loss‚40.49552407972392 /loss/FRt1/assignment_loss‚2.794045873663642 1loss/SRqqt2/detection_loss‚2.2340547160522357 /loss/FBt4/detection_loss‚0.5932236321831262 1loss/SRqqt1/assignment_loss‚1.292762692958275 2loss/SRqqt4/assignment_loss‚0.9982282230487236 1loss/FBt2/assignment_loss‚0.19985983660710252Éê©c»(Ý % _timestamp‚1.7884253395167952e+09 1loss/FBt4/assignment_loss‚0.12062941262459664 .loss/FRt2/detection_loss‚2.979726036389669 /loss/FRt3/detection_loss‚2.4206886386329476 0loss/FRt2/assignment_loss‚3.3760253768978696 0loss/FRt4/assignment_loss‚1.4773745328187942 1loss/SRqqt1/detection_loss‚1.8598497742985158 /loss/FBt2/detection_loss‚2.5914827035023618 /loss/FRt4/detection_loss‚1.6666184117396672 1loss/SRqqt4/detection_loss‚1.0273831586425122 _runtime‚8839.254262463 1loss/SRqqt2/assignment_loss‚1.800287685432547 2loss/SRqqt3/assignment_loss‚1.4643788637839399 1loss/FBt1/assignment_loss‚0.18458679349472126 %loss/total_loss‚40.49552407972392 /loss/FRt1/assignment_loss‚2.794045873663642 1loss/SRqqt2/detection_loss‚2.2340547160522357 /loss/FBt4/detection_loss‚0.5932236321831262 1loss/SRqqt1/assignment_loss‚1.292762692958275 2loss/SRqqt4/assignment_loss‚0.9982282230487236 1loss/FBt2/assignment_loss‚0.19985983660710252 1loss/SRqqt3/detection_loss‚1.7565697993169416 /loss/FBt1/detection_loss‚2.6877306832207575 trainer/global_step‚11049 _step‚910 0loss/FRt3/assignment_loss‚2.1208296418190002 epoch‚9 1loss/FBt3/assignment_loss‚0.06564388733305451 .loss/FRt1/detection_loss‚2.290331661023877 .loss/FBt3/detection_loss‚2.493212044239044ŽúqñS¼(‚(:’ †éäÔ…ÙŒ~ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.504999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/G`Es½(‚(:è †éäÔ¿Ý­€ disk.dm-1.in‚ 126.77734375 network.sent‚ 2.6898672e+07 disk.drbd1716.in‚0% disk.nvme0n1.out‚38065.9736328125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0# memory_percent‚94.2040205001831 disk.nvme0n1.in‚4.6875 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 464.42578125" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.sda2.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚38065.9736328125 disk.drbd2266.in‚0 disk.sda1.out‚0 cpu‚25.357167926693577' disk.nvme0c0n1.out‚38065.9736328125 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda3.out‚ 464.42578125 network.recv‚ 3.208661e+06 disk.dm-1.out‚ 464.42578125 disk.sda1.in‚0( proc.memory.availableMB‚ 949.61328125* disk./.usagePercent‚17.526602971719075 disk.rbd0.in‚0 disk.drbd2266.out‚0% disk./.usageGB‚261.15904998779297 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375a·ct’¾(‚(:‡ •éäÔ†§~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­µÆÛp¿(‚(:å •éäÔö½€ disk.drbd2266.out‚0 disk.dm-1.out‚ 464.91796875 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.sda3.out‚ 464.91796875 disk.rbd1.in‚0 disk.sda.out‚ 464.91796875$ memory_percent‚94.20256614685059' proc.memory.availableMB‚ 949.8515625* proc.memory.percent‚16.921377182006836 network.sent‚ 2.7007147e+07 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.599609375 disk.sda2.in‚0 cpu‚25.35662084783963 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875$ disk./.usageGB‚261.1756782531738 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375) disk./.usagePercent‚17.52771890856083 disk.drbd2080.out‚0" disk.dm-0.out‚38119.5439453125 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.6875' disk.nvme0c0n1.out‚38119.5439453125 network.recv‚ 3.214561e+06 disk.dm-0.in‚4.6875% disk.nvme0n1.out‚38119.5439453125 disk.rbd0.in‚0(Ùñ½2À(- _wandbruntime‚8874  _runtime‚8874Á1Í ÈÁ( & lr-AdamW/pg1‚3.428981573866786e-06 & lr-AdamW/pg2‚3.428981573866786e-06 trainer/global_step‚11099 % _timestamp‚1.7884253750253487e+09 _runtime‚8874.762983155 _step‚911EhtžÍÂ(Ç & lr-AdamW/pg1‚3.428981573866786e-06 & lr-AdamW/pg2‚3.428981573866786e-06 trainer/global_step‚11099 % _timestamp‚1.7884253750253487e+09 _runtime‚8874.762983155 _step‚911\®h÷2Ã(- _wandbruntime‚8874  _runtime‚8874‹®¢Fè'Ä(âO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305777707520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195418 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209755663 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147522764 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161712198 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9760660578816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161712200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306434070016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705819360338432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209735625 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705819384278528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655724704098304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580221064192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9760836785664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53164496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704098304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147523232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53164962 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94358270 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚638734607991774 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 % _timestamp‚1.7884253750260713e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2yŒ4×Å(çO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209755663 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚705819384278528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚655724704098304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161712198 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚655724029732864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚646418270028288 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚646418252025344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚715580221064192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9760836785664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53164496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚655724704098304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371467863 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147523232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53164962 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94358270 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚638734607991774 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚629463094234112 % _timestamp‚1.7884253750260713e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209735625 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305777707520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638735279468890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9272168195418 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306434070016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209755663 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147522764 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161712198 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161712200 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚646418270028288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚466 ?9DeviceStatsMoÞfæ nitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9271513757662 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚468 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 200146944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9760660578816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161712200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94358268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306434070016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚705819360338432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209735625 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655781202199552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646474623067136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209754079 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884253757026691e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597135360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53169477 trainer/global_step‚11099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161726214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754079 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147535465 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9272984630590 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655781876578304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629518015404032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307253511168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147535915 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53169029 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307253511168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9272984630590 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚638791000034622 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 371500359 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚655781876578304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚629518015404032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚450 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307253511168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147535915 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 _step‚913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚646474605064192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚629517998364672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53169029 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94366436 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371500359 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚655781202199552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚646474623067136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚646474623067136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209754079 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884253757026691e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9306597135360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚705881035016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53169477 trainer/global_step‚11099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161726214 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 209754079 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9306597135360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147535465 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371480293 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚646474605064192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚705881011076096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0‘õ.2É(- _wandbruntime‚8875  _runtime‚8875>L§mYÊ(Ó 0loss/FRt2/assignment_loss‚2.5197212238644444 2loss/SRqqt3/assignment_loss‚1.4593665501408613 2loss/SRqqt1/assignment_loss‚1.4217733442783356 2loss/SRqqt2/assignment_loss‚1.7716841456981807 .loss/FRt4/detection_loss‚1.583644721582532 0loss/FBt4/detection_loss‚0.47020899821796175 /loss/FRt3/assignment_loss‚2.413909768571659 .loss/FRt2/detection_loss‚2.427918471569239 _step‚914 0loss/FBt2/assignment_loss‚0.5349608048796654 0loss/FBt3/assignment_loss‚0.1322534759267662 1loss/SRqqt4/detection_loss‚1.2897899197747134 % _timestamp‚1.7884253757045348e+09 2loss/SRqqt4/assignment_loss‚0.7661410228881452 1loss/FBt1/assignment_loss‚0.14241155640532574 .loss/FBt2/detection_loss‚3.409023571014404 &loss/total_loss‚41.058550859502205 _runtime‚8875.442001376 0loss/FRt4/assignment_loss‚1.1956629073619842 .loss/FRt1/detection_loss‚2.557257859871305 .loss/FRt3/detection_loss‚2.207436037763041 1loss/SRqqt1/detection_loss‚1.8713687055490233 1loss/SRqqt3/detection_loss‚1.4535118506025912 trainer/global_step‚11099 /loss/FRt1/assignment_loss‚4.023357453017399 -loss/FBt3/detection_loss‚2.18272220546549 epoch‚9 0loss/FBt4/assignment_loss‚0.0832519659739004 0loss/SRqqt2/detection_loss‚2.209009219438602 /loss/FBt1/detection_loss‚2.9321650796466403è\‘^Ë(Ø 0loss/FRt2/assignment_loss‚2.5197212238644444 2loss/SRqqt3/assignment_loss‚1.4593665501408613 2loss/SRqqt1/assignment_loss‚1.4217733442783356 2loss/SRqqt2/assignment_loss‚1.7716841456981807 .loss/FRt4/detection_loss‚1.583644721582532 0loss/FBt4/detection_loss‚0.47020899821796175 /loss/FRt3/assignment_loss‚2.413909768571659 .loss/FRt2/detection_loss‚2.427918471569239 _step‚914 0loss/FBt2/assignment_loss‚0.5349608048796654 0loss/FBt3/assignment_loss‚0.1322534759267662 1loss/SRqqt4/detection_loss‚1.2897899197747134 % _timestamp‚1.7884253757045348e+09 2loss/SRqqt4/assignment_loss‚0.7661410228881452 1loss/FBt1/assignment_loss‚0.14241155640532574 .loss/FBt2/detection_loss‚3.409023571014404 &loss/total_loss‚41.058550859502205 _runtime‚8875.442001376 0loss/FRt4/assignment_loss‚1.1956629073619842 .loss/FRt1/detection_loss‚2.557257859871305 .loss/FRt3/detection_loss‚2.207436037763041 1loss/SRqqt1/detection_loss‚1.8713687055490233 1loss/SRqqt3/detection_loss‚1.4535118506025912 trainer/global_step‚11099 /loss/FRt1/assignment_loss‚4.023357453017399 -loss/FBt3/detection_loss‚2.18272220546549 epoch‚9 0loss/FBt4/assignment_loss‚0.0832519659739004 0loss/SRqqt2/detection_loss‚2.209009219438602 /loss/FBt1/detection_loss‚2.9321650796466403’ÿ£Žô’Ì(‚(:‡ ¤éäÔí¾¼ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.431' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›lemÍ(‚(:â ¤éäÔÎ×Ý€ disk.rbd1.out‚ 4.599609375 disk.sda2.out‚0 disk.drbd2080.in‚0 network.sent‚ 2.7013621e+07 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875 network.recv‚ 3.219667e+06 disk.dm-1.out‚ 465.46875 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.20285224914551 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.out‚ 465.46875* proc.memory.percent‚16.921377182006836 cpu‚25.356074895402323( disk.nvme0c0n1.out‚38156.86376953125% disk./.usageGB‚261.18893814086914 disk.dm-0.in‚4.6875# disk.dm-0.out‚38156.86376953125 disk.sda.in‚ 126.77734375 disk.sda.out‚ 465.46875" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚38156.86376953125 disk.drbd2265.out‚0* disk./.usagePercent‚17.528608790749843 disk.sda2.in‚0' proc.memory.availableMB‚ 949.8046875²=xá’Î(‚(:‡ ³éäԬɳ~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚157.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚31.4112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sd;vsÏ(‚(:è ³éäÔü„ž€ disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.rbd1.out‚ 4.599609375' proc.memory.availableMB‚ 949.8203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚261.4517707824707 disk.sda.in‚ 126.77734375 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 network.sent‚ 2.712155e+07 disk.sda1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚38213.20654296875 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 3.225019e+06 disk.rbd0.in‚0 disk.sda3.out‚ 466.00390625 disk.drbd2265.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚38213.20654296875 disk.dm-1.out‚ 466.00390625 disk.drbd2266.out‚0* disk./.usagePercent‚17.546247709859003( disk.nvme0c0n1.out‚38213.20654296875$ memory_percent‚94.20275688171387" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 cpu‚25.35559120197778 proc.cpu.threads‚17 disk.sda.out‚ 466.00390625 disk.nvme0n1.in‚4.6875 Ûa”’Ð(‚(:‡ ÂéäÔ¯¾¦~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ^áÄlÑ(‚(:á ÂéäÔå´—€ disk.sda.out‚ 466.640625 disk.sda1.out‚0* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17 disk.sda3.out‚ 466.640625 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚261.4598197937012 disk.sda2.out‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚38283.34423828125 disk.rbd1.in‚0 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚38283.34423828125$ memory_percent‚94.19851303100586 network.sent‚ 2.7127669e+07 network.recv‚ 3.229535e+06 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 466.640625 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 disk.rbd1.out‚ 4.599609375 disk.drbd1716.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚17.54678788575625 disk.sda2.in‚0 disk.drbd2265.in‚0# disk.dm-0.out‚38283.34423828125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.drbd2266.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0& proc.memory.availableMB‚ 950.515625 cpu‚25.35505059218624 disk.drbd2265.out‚0Õâ 2Ò(- _wandbruntime‚8911  _runtime‚8911O‰ÞcÈÓ( & lr-AdamW/pg2‚3.093776266239161e-06 trainer/global_step‚11149 % _timestamp‚1.7884254119230385e+09 _runtime‚8911.660574079 _step‚915 & lr-AdamW/pg1‚3.093776266239161e-06qìÍÔ(Ç & lr-AdamW/pg1‚3.093776266239161e-06 & lr-AdamW/pg2‚3.093776266239161e-06 trainer/global_step‚11149 % _timestamp‚1.7884254119230385e+09 _runtime‚8911.660574079 _step‚915“îˆ2Õ(- _wandbruntime‚8911  _runtime‚8911„v¹Ö(âO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 162412998 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9803686203392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9313003534826 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _runtime‚8911.662031569 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162413000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 210661677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210681715 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 373094715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚455 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336298 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763513344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9803856118784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210661677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162413000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074675 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718705777280512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763513344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 trainer/global_step‚11149 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649235921970688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148159790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658583341846528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341846528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53393120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649235921970688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94766670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚641521321336298 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9346763513344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9803856118784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210661677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649235903967744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162413000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649235903967744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 373074675 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚916 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9312349097070 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658582667481088 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚708901921161728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚718705777280512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9346763513344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚632208317801472 trainer/global_step‚11149 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649235921970688 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148159790 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚457 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94766668 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 193855488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658583341846528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658583341846528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53393120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649235921970688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94766670 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9804714201600 trainer/global_step‚11149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53397583 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94774836 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚8912.355623621 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171985 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768286100992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚917 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210700195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53397149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436OzC‚Ú(ðM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9348239316992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148171985 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9313165532034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚718768286100992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚917 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 210680131 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210700195 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53397149 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 $ _timestamp‚1.788425412617429e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9313819969998 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658640514326528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649292275009536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9804714201600 trainer/global_step‚11149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658640514326528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9348239316992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53397583 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162427014 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 373127211 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚632263221932032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 373107145 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚94774836 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚632263204892672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9347582941184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚649292257006592 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658639839947776 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162427016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚8912.355623621•h‹ý2Û(- _wandbruntime‚8912  _runtime‚8912í†~}ZÜ(Ô 1loss/SRqqt3/detection_loss‚1.7775996400500245 epoch‚9 0loss/FRt1/assignment_loss‚3.4576910876709483 1loss/SRqqt1/assignment_loss‚1.415456978713765 2loss/SRqqt3/assignment_loss‚1.4685105625188575 2loss/SRqqt4/assignment_loss‚0.8627762452057484 /loss/FBt3/detection_loss‚3.4106313486893973 2loss/SRqqt2/assignment_loss‚1.5125147716491483 1loss/FBt2/assignment_loss‚0.22898723825267098 1loss/FBt3/assignment_loss‚0.22103511497001213 0loss/SRqqt1/detection_loss‚2.019212283340155 %loss/total_loss‚43.32006093657813 % _timestamp‚1.7884254126192532e+09 _runtime‚8912.356702407 /loss/FRt3/assignment_loss‚2.864885601012603 /loss/FRt2/detection_loss‚3.1935577919849982 -loss/FRt3/detection_loss‚2.37696141263713 trainer/global_step‚11149 .loss/FRt4/detection_loss‚1.797653948546064 1loss/SRqqt4/detection_loss‚0.9404970458236294 /loss/FRt1/detection_loss‚2.2110383491153303 /loss/FBt1/detection_loss‚2.2262633164723713 /loss/FRt4/assignment_loss‚1.536251811348662 0loss/SRqqt2/detection_loss‚2.423564752191305 .loss/FBt2/detection_loss‚3.013085573911667 /loss/FBt4/detection_loss‚0.5846958090723665 0loss/FRt2/assignment_loss‚3.3595838157030253 1loss/FBt4/assignment_loss‚0.15277470927213102 _step‚918 0loss/FBt1/assignment_loss‚0.2648317284261168É õD_Ý(Ù 2loss/SRqqt2/assignment_loss‚1.5125147716491483 1loss/FBt2/assignment_loss‚0.22898723825267098 1loss/FBt3/assignment_loss‚0.22103511497001213 0loss/SRqqt1/detection_loss‚2.019212283340155 %loss/total_loss‚43.32006093657813 % _timestamp‚1.7884254126192532e+09 _runtime‚8912.356702407 /loss/FRt3/assignment_loss‚2.864885601012603 /loss/FRt2/detection_loss‚3.1935577919849982 -loss/FRt3/detection_loss‚2.37696141263713 trainer/global_step‚11149 .loss/FRt4/detection_loss‚1.797653948546064 1loss/SRqqt4/detection_loss‚0.9404970458236294 /loss/FRt1/detection_loss‚2.2110383491153303 /loss/FBt1/detection_loss‚2.2262633164723713 /loss/FRt4/assignment_loss‚1.536251811348662 0loss/SRqqt2/detection_loss‚2.423564752191305 .loss/FBt2/detection_loss‚3.013085573911667 /loss/FBt4/detection_loss‚0.5846958090723665 0loss/FRt2/assignment_loss‚3.3595838157030253 1loss/FBt4/assignment_loss‚0.15277470927213102 _step‚918 0loss/FBt1/assignment_loss‚0.2648317284261168 1loss/SRqqt3/detection_loss‚1.7775996400500245 epoch‚9 0loss/FRt1/assignment_loss‚3.4576910876709483 1loss/SRqqt1/assignment_loss‚1.415456978713765 2loss/SRqqt3/assignment_loss‚1.4685105625188575 2loss/SRqqt4/assignment_loss‚0.8627762452057484 /loss/FBt3/detection_loss‚3.4106313486893973–4Þ(‚(:„ ÑéäÔâ—•~ gpu.0.gpu‚27.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚35.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07tfuß(‚(:ê ÑéäÔŸŒ£€ disk.dm-0.in‚4.6875( disk.nvme0c0n1.out‚38326.70361328125 disk.sda3.in‚ 126.77734375$ disk./.usageGB‚261.4643974304199 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.dm-1.out‚ 467.05859375 disk.rbd1.in‚0 disk.sda.out‚ 467.05859375 disk.sda1.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚38326.70361328125 disk.sda2.out‚0 disk.sda3.out‚ 467.05859375# disk.dm-0.out‚38326.70361328125 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0* proc.memory.percent‚16.921377182006836 cpu‚25.354566770113763 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.recv‚ 3.233723e+06) disk./.usagePercent‚17.54709509479734 disk.sda2.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 950.45703125" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.7133534e+07 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2080.out‚0$ memory_percent‚94.19887065887451 Ì`à(‚(:’ àéäÔ¼ñˆ~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.319' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.263799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ jrá(‚(:ç àéäÔâ“€ disk.dm-1.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 cpu‚25.354030105846164% disk./.usageGB‚261.48242568969727 disk.rbd1.out‚ 4.599609375 disk.sda1.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 467.70703125 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0% disk.nvme0n1.out‚38379.0595703125 disk.sda2.in‚0$ memory_percent‚94.20042037963867 network.sent‚ 2.7242372e+07 disk.drbd2265.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2772.3984375* disk./.usagePercent‚17.548304986404155 disk.rbd0.out‚0& proc.memory.availableMB‚ 950.203125' disk.nvme0c0n1.out‚38379.0595703125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 network.recv‚ 3.240032e+06" disk.dm-0.out‚38379.0595703125 disk.sda2.out‚0 disk.sda3.out‚ 467.70703125 disk.sda.out‚ 467.70703125* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17ã‘g2â(- _wandbruntime‚8948  _runtime‚8948us¢dÊã(Ä _runtime‚8948.399458989 _step‚919 ' lr-AdamW/pg1‚2.7755032382845332e-06 ' lr-AdamW/pg2‚2.7755032382845332e-06 trainer/global_step‚11199 % _timestamp‚1.7884254486619298e+09®´EÝÏä(É ' lr-AdamW/pg2‚2.7755032382845332e-06 trainer/global_step‚11199 % _timestamp‚1.7884254486619298e+09 _runtime‚8948.399458989 _step‚919 ' lr-AdamW/pg1‚2.7755032382845332e-06—P 2å(- _wandbruntime‚8948  _runtime‚8948úpZç'æ(áO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831292860416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405681664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652053573913088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 trainer/global_step‚11199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53621274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607767 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634953507290112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644306691726590 $ _timestamp‚1.788425448662762e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53620839 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587729 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9846858755584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˆ d,H'ç(æO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚661441305229312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚435 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚652053573913088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚721831292860416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9388405681664 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211607767 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 179175424 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652053555910144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚652053573913088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95175070 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634953524329472 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9353184436478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚711984458044928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 trainer/global_step‚11199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53621274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211607767 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211587729 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634953507290112 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 155235328 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚644306691726590 $ _timestamp‚1.788425448662762e+09 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53620839 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _step‚920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9353838874234 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211587729 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9846858755584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚661441305229312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374721567 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.þ?¿M¤on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944˜J¯!x2è(- _wandbruntime‚8949  _runtime‚8949,üÂïò&é(ìM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148808610 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661499152074752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9388568747008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354655309406 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163127816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53625374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635008428460032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721893808175104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721893993628672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884254495809524e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9847723332608 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚921 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53625820 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389225122816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚11199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661498477696000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606183 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225122816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9847884846080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8949.319301971 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374754063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53625820 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9389225122816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚448 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163127814 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9388568747008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163127816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 trainer/global_step‚11199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211606183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374733997 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚661498477696000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211606183 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚712046108782592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652109908948992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9389225122816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9847884846080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚8949.319301971 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 185453568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 161513472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148808610 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211626247 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661499152074752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95183236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9388568747008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚652109926951936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9354655309406 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 163127816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95183238 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652109908948992 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 374754063 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652109926951936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53625374 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚635008428460032 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721893808175104 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721893993628672 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚635008411420672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884254495809524e+09™>ظ22ë(- _wandbruntime‚8949  _runtime‚8949ÿå@Ì\ì(Ö _runtime‚8949.320532032 1loss/FBt1/assignment_loss‚0.43597778791028724 -loss/FBt1/detection_loss‚2.81171007667269 /loss/FBt2/detection_loss‚2.3319689565234714 2loss/SRqqt3/assignment_loss‚1.3590986480703577 1loss/SRqqt3/detection_loss‚1.6630965676158667 &loss/total_loss‚43.508169193088364 /loss/FRt3/assignment_loss‚2.479780543682187 0loss/FRt1/assignment_loss‚3.0180404742558795 epoch‚9 trainer/global_step‚11199 2loss/SRqqt1/assignment_loss‚1.4638304173384074 2loss/SRqqt4/assignment_loss‚1.1454187820555042 0loss/FBt3/assignment_loss‚0.7253331210464239 1loss/FBt4/assignment_loss‚0.14197459921801628 /loss/FBt4/detection_loss‚0.6783340506471421 0loss/FRt4/assignment_loss‚1.6653462861265456 1loss/SRqqt2/assignment_loss‚2.003295282522837 0loss/SRqqt2/detection_loss‚2.177830549803647 /loss/FBt3/detection_loss‚3.0636628031730653 /loss/FRt2/assignment_loss‚4.157169505169517 1loss/FBt2/assignment_loss‚0.13988489080737862 0loss/SRqqt4/detection_loss‚1.052939668108346 % _timestamp‚1.7884254495830534e+09 _step‚922 /loss/FRt1/detection_loss‚2.2678939322630565 .loss/FRt2/detection_loss‚2.640516227797458 /loss/FRt3/detection_loss‚2.3569690262855487 /loss/FRt4/detection_loss‚1.8394425362348557 1loss/SRqqt1/detection_loss‚1.8886544597598742û]û¯aí(Û 0loss/FRt1/assignment_loss‚3.0180404742558795 epoch‚9 trainer/global_step‚11199 2loss/SRqqt1/assignment_loss‚1.4638304173384074 2loss/SRqqt4/assignment_loss‚1.1454187820555042 0loss/FBt3/assignment_loss‚0.7253331210464239 1loss/FBt4/assignment_loss‚0.14197459921801628 /loss/FBt4/detection_loss‚0.6783340506471421 0loss/FRt4/assignment_loss‚1.6653462861265456 1loss/SRqqt2/assignment_loss‚2.003295282522837 0loss/SRqqt2/detection_loss‚2.177830549803647 /loss/FBt3/detection_loss‚3.0636628031730653 /loss/FRt2/assignment_loss‚4.157169505169517 1loss/FBt2/assignment_loss‚0.13988489080737862 0loss/SRqqt4/detection_loss‚1.052939668108346 % _timestamp‚1.7884254495830534e+09 _step‚922 /loss/FRt1/detection_loss‚2.2678939322630565 .loss/FRt2/detection_loss‚2.640516227797458 /loss/FRt3/detection_loss‚2.3569690262855487 /loss/FRt4/detection_loss‚1.8394425362348557 1loss/SRqqt1/detection_loss‚1.8886544597598742 _runtime‚8949.320532032 1loss/FBt1/assignment_loss‚0.43597778791028724 -loss/FBt1/detection_loss‚2.81171007667269 /loss/FBt2/detection_loss‚2.3319689565234714 2loss/SRqqt3/assignment_loss‚1.3590986480703577 1loss/SRqqt3/detection_loss‚1.6630965676158667 &loss/total_loss‚43.508169193088364 /loss/FRt3/assignment_loss‚2.479780543682187š˜¶i’î(‚(:‡ ïéäÔ¯³’ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š9rrï(‚(:ç ïéäÔ¶‘À€ disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 468.15234375 disk.sda2.out‚0 disk.dm-0.in‚4.6875" disk.dm-0.out‚38432.9033203125 disk.sda.out‚ 468.15234375 disk.sda1.out‚0 disk.drbd1716.out‚0% disk./.usageGB‚261.50472259521484 disk.sda.in‚ 126.77734375 disk.rbd0.out‚0% disk.nvme0n1.out‚38432.9033203125 network.sent‚ 2.7246729e+07 network.recv‚ 3.242595e+06 disk.sda1.in‚0 disk.drbd1716.in‚0' disk.nvme0c0n1.out‚38432.9033203125$ memory_percent‚94.19910907745361 proc.cpu.threads‚17 disk.drbd2266.in‚0( proc.memory.availableMB‚ 950.41796875 disk.sda3.out‚ 468.15234375 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚4.6875* proc.memory.percent‚16.921377182006836 cpu‚25.35354807993964 disk.drbd2080.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0) disk./.usagePercent‚17.54980135044179 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375" proc.memory.rssMB‚ 2772.3984375ýôx‘ð(‚(:† þéäÔúÌÿ} gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚179.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰½7oñ(‚(:ä þéäÔ™Ëÿ€& disk.nvme0c0n1.out‚38505.150390625) disk./.usagePercent‚17.55035560675342 disk.rbd1.out‚ 4.599609375 disk.sda.out‚ 468.83203125 disk.nvme0c0n1.in‚4.6875$ memory_percent‚94.19910907745361( proc.memory.availableMB‚ 950.41796875 network.recv‚ 3.249179e+06! disk.dm-0.out‚38505.150390625 disk.dm-1.out‚ 468.83203125 disk.rbd0.out‚0 network.sent‚ 2.7355624e+07$ disk./.usageGB‚261.5129814147949 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda2.out‚0" proc.memory.rssMB‚ 2772.3984375 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0$ disk.nvme0n1.out‚38505.150390625 disk.sda2.in‚0 disk.sda3.out‚ 468.83203125 cpu‚25.353011324737597 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚00ËÁŽò(‚(:… êäÔðñ~ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.48' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó{§Fpó(‚(:å êäÔêèµ€ disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 network.sent‚ 2.7361445e+07& disk.nvme0n1.out‚38555.39501953125 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 network.recv‚ 3.253236e+06 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.dm-1.out‚ 469.203125 disk.drbd1716.in‚0$ memory_percent‚94.19910907745361* proc.memory.percent‚16.921377182006836( disk.nvme0c0n1.out‚38555.39501953125 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375" proc.memory.rssMB‚ 2772.3984375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 469.203125( proc.memory.availableMB‚ 950.41796875 cpu‚25.352482499751616 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 469.203125) disk./.usagePercent‚17.53839467874618% disk./.usageGB‚261.33475494384766 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚38555.39501953125 disk.drbd2266.in‚0§›´P2ô(- _wandbruntime‚8985  _runtime‚8985ÿ]DéÊõ(Ä trainer/global_step‚11249 % _timestamp‚1.7884254857221014e+09 _runtime‚8985.459610229 _step‚923 ' lr-AdamW/pg1‚2.4742288176248763e-06 ' lr-AdamW/pg2‚2.4742288176248763e-06Ê‹”Ïö(É ' lr-AdamW/pg1‚2.4742288176248763e-06 ' lr-AdamW/pg2‚2.4742288176248763e-06 trainer/global_step‚11249 % _timestamp‚1.7884254857221014e+09 _runtime‚8985.459610229 _step‚923›Óì-2÷(- _wandbruntime‚8985  _runtime‚8985*tVÓø(áO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53848568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚724957024521216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53848993 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724956836957184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212513781 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212533819 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚ $|844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚654871225855488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889865969152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942977536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149432463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376348419 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149432036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890029593088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617342976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328379 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391487488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 $ _timestamp‚1.788425485722795e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PWì'ù(æO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664300617342976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 _step‚924 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚654871207852544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53848568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95583468 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚715066994928128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚715066970988032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163814600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚724957024521216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 163814598 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 trainer/global_step‚11249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664299942977536 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53848993 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚724956836957184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 212513781 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163814598 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212533819 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚654871225855488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889865969152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚637698730857472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664299942977536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9429391487488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163814600 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212513781 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647092733593998 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149432463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚425 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚654871207852544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376348419 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149432036 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9890029593088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664300617342976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376328379 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9429391487488 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 $ _timestamp‚1.788425485722795e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212533819 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚8986.162798989 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9430210928640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9429554552832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53853062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884254864248571e+09 _step‚925 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212532235 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149444698 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019341855744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664357115444224 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9430210928640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9429554552832 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 163828616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚664357789822976 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9429554552832 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚654927578894336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 212532235 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53853062 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 % _timestamp‚1.7884254864248571e+09 _step‚925 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212532235 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149444698 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725019341855744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376380915 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚715128645665792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 _runtime‚8986.162798989 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376380915 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 P eStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚654927560891392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163828614 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9430210928640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9890887934976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394836210850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95591636 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 163828614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212552299 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844ÚÅ«H2ý(- _wandbruntime‚8986  _runtime‚8986áÔZþ(Ô _step‚926 $ _timestamp‚1.788425486426808e+09 1loss/FBt4/assignment_loss‚0.12289410622448328 .loss/FRt1/detection_loss‚2.598865203346525 /loss/FRt3/detection_loss‚2.3391867200178758 /loss/FBt1/detection_loss‚2.1614342510700224 /loss/FBt2/detection_loss‚2.7481011748313904 /loss/FRt2/assignment_loss‚3.352907262245814 2loss/SRqqt4/assignment_loss‚0.9082280367597817 1loss/FBt3/assignment_loss‚0.11133043104198255 0loss/FRt3/assignment_loss‚2.8868856728076935 1loss/SRqqt3/assignment_loss‚1.556676040439365 .loss/FRt2/detection_loss‚2.923556975523631 /loss/FBt3/detection_loss‚3.1771916647752128 2loss/SRqqt1/assignment_loss‚1.2844955480202687 1loss/FBt1/assignment_loss‚0.33408548263832927 0loss/SRqqt3/detection_loss‚1.762651045090304 1loss/SRqqt1/detection_loss‚1.9563365460828293 %loss/total_loss‚42.80193423427244 epoch‚9 trainer/global_step‚11249 /loss/FBt4/detection_loss‚0.5243661945538693 0loss/FRt1/assignment_loss‚2.6280985474586487 /loss/FBt2/assignment_loss‚0.448294441960752 /loss/FRt4/detection_loss‚1.5518982791483908 1loss/SRqqt2/detection_loss‚2.4835198185136242 1loss/SRqqt4/detection_loss‚1.0966945934079346 0loss/FRt4/assignment_loss‚1.7596632312920133 1loss/SRqqt2/assignment_loss‚2.084572967021696 _runtime‚ 8986.16426743,¤íŠ_ÿ(Ù 2loss/SRqqt1/assignment_loss‚1.2844955480202687 1loss/FBt1/assignment_loss‚0.33408548263832927 0loss/SRqqt3/detection_loss‚1.762651045090304 1loss/SRqqt1/detection_loss‚1.9563365460828293 %loss/total_loss‚42.80193423427244 epoch‚9 trainer/global_step‚11249 /loss/FBt4/detection_loss‚0.5243661945538693 0loss/FRt1/assignment_loss‚2.6280985474586487 /loss/FBt2/assignment_loss‚0.448294441960752 /loss/FRt4/detection_loss‚1.5518982791483908 1loss/SRqqt2/detection_loss‚2.4835198185136242 1loss/SRqqt4/detection_loss‚1.0966945934079346 0loss/FRt4/assignment_loss‚1.7596632312920133 1loss/SRqqt2/assignment_loss‚2.084572967021696 _runtime‚ 8986.16426743 _step‚926 $ _timestamp‚1.788425486426808e+09 1loss/FBt4/assignment_loss‚0.12289410622448328 .loss/FRt1/detection_loss‚2.598865203346525 /loss/FRt3/detection_loss‚2.3391867200178758 /loss/FBt1/detection_loss‚2.1614342510700224 /loss/FBt2/detection_loss‚2.7481011748313904 /loss/FRt2/assignment_loss‚3.352907262245814 2loss/SRqqt4/assignment_loss‚0.9082280367597817 1loss/FBt3/assignment_loss‚0.11133043104198255 0loss/FRt3/assignment_loss‚2.8868856728076935 1loss/SRqqt3/assignment_loss‚1.556676040439365 .loss/FRt2/detection_loss‚2.923556975523631 /loss/FBt3/detection_loss‚3.1771916647752128žQr¶ã€)‚(:… œêäÔå´‹~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.18' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Ê‹‰u)‚(:ê œêäÔ…²¦€ disk.rbd1.out‚ 4.599609375 disk.sda.in‚ 126.77734375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2080.in‚0$ memory_percent‚94.20254230499268 disk.nvme0n1.in‚4.6875( proc.memory.availableMB‚ 949.85546875" proc.memory.rssMB‚ 2772.3984375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.7367777e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.out‚ 469.76171875 disk.rbd0.out‚0 network.recv‚ 3.25797e+06* disk./.usagePercent‚17.549514365995908 disk.drbd1716.out‚0& disk.nvme0n1.out‚38626.48388671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0# disk.dm-0.out‚38626.48388671875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 469.76171875 cpu‚25.351980532912805$ disk./.usageGB‚261.5004463195801 disk.sda1.out‚0 disk.sda3.out‚ 469.76171875* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚38626.48388671875 disk.dm-0.in‚4.6875ßø„Ò‚)‚(:’ «êäԚˆ~ gpu.0.gpu‚49.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.291' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.458199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÄÐ8vƒ)‚(:ë «êäÔü±Þ€ network.recv‚ 3.263403e+06 network.sent‚ 2.7475702e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 cpu‚25.351476999563975 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0' proc.memory.availableMB‚ 950.6015625 disk.drbd2265.out‚0# disk.dm-0.out‚38675.43701171875 disk.sda.out‚ 470.15234375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 470.15234375* disk./.usagePercent‚17.551459767248602 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.599609375 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚38675.43701171875$ memory_percent‚94.19798851013184 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 470.15234375 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375% disk./.usageGB‚261.52943420410156 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17 disk.drbd2080.in‚0& disk.nvme0n1.out‚38675.43701171875 disk.nvme0c0n1.in‚4.6875* proc.memory.percent‚16.921377182006836ÅITÖ2„)- _wandbruntime‚9022  _runtime‚9022ÐLêVÈ…) & lr-AdamW/pg1‚2.190015789397899e-06 & lr-AdamW/pg2‚2.190015789397899e-06 trainer/global_step‚11299 % _timestamp‚1.7884255226645255e+09 _runtime‚9022.402124418 _step‚927öÝÒ)͆)Ç % _timestamp‚1.7884255226645255e+09 _runtime‚9022.402124418 _step‚927 & lr-AdamW/pg1‚2.190015789397899e-06 & lr-AdamW/pg2‚2.190015789397899e-06 trainer/global_step‚11299Ÿ±>Íœ2‡)- _wandbruntime‚9022  _runtime‚9022#o_bè'ˆ)âO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54076817 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150069131 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54077261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657688859794944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657688859794944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213459871 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938522 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95991870 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9022.403108114 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728082393084416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975271 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213439833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553050 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚667159255091200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164515400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884255226652665e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657688877797888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377293312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991868 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461406 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657688877797888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9932885213184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255091200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580725760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150068685 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728082582745600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11299 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9933050934272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0®^üí'‰)çO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164515400 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150068685 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚728082582745600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚11299 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9434855115294 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9933050934272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377955231 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213439833 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9470377293312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚667158580725760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9469720930816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54076817 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚640443920346112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164515398 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _step‚928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150069131 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54077261 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657688859794944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657688859794944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213459871 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚649879446938522 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚95991870 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚9022.403108114 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718149507871232 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚728082393084416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 377975271 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213439833 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚444 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9435509553050 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164515398 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚667159255091200 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213459871 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 164515400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚718149531811328 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 % _timestamp‚1.7884255226652665e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚657688877797888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9470377293312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95991868 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚446 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚649878775461406 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657688877797888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9932885213184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚667159255091200 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚667158580725760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ãÁáð2Š)- _wandbruntime‚9023  _runtime‚9023ÐRO ‹)ìM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377987701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚9023.091145582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478351 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚929 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478351 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9933910003200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504254 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164529414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9933746392576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚11299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667216427571200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150081394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54081358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚728144905001472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884255233530061e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150081846 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚667216427571200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0sq$Ý÷&Œ)ñM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377987701 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213458287 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚649934496026930 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚9023.091145582 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213478351 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 _step‚929 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 378007767 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213478351 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9470540358656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9471196734464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96000038 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9933910003200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚649935167504254 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9436325988222 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 164529416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚640498824476672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164529414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9933746392576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 trainer/global_step‚11299 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9435671550258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 377987701 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚667216427571200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150081394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚640498841516032 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚657745230836736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9471196734464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378007767 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54081358 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚728144905001472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 % _timestamp‚1.7884255233530061e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9470540358656 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150081846 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚667215753192448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚667216427571200 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657745212833792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚452¡‹ê2)- _wandbruntime‚9023  _runtime‚9023͉ì:VŽ)Ð 1loss/SRqqt4/detection_loss‚1.5390173014180333 /loss/FBt2/detection_loss‚2.1271625638008116 /loss/FBt4/detection_loss‚0.7755953838738302 1loss/FBt4/assignment_loss‚0.06252646587748029 2loss/SRqqt3/assignment_loss‚1.3909269191324711 2loss/SRqqt4/assignment_loss‚0.9335651821548155 /loss/FBt1/assignment_loss‚2.309713721740991 epoch‚9 /loss/FRt1/detection_loss‚2.1343013131618505 _step‚930 &loss/total_loss‚45.939079851983124 0loss/FRt2/assignment_loss‚3.3550243148436913 /loss/FRt3/assignment_loss‚2.881617928450962 .loss/FBt2/assignment_loss‚0.11094804746308 .loss/FRt3/detection_loss‚2.178065041044973 /loss/FRt4/detection_loss‚1.8368129450629682 1loss/SRqqt1/detection_loss‚1.8180107030333306 1loss/SRqqt3/detection_loss‚1.5262792551746736 .loss/FRt1/assignment_loss‚3.20776914358139 0loss/FRt4/assignment_loss‚1.7257321400845305 .loss/FBt1/detection_loss‚5.757795989513397 % _timestamp‚1.7884255233550737e+09 _runtime‚9023.092534764 2loss/SRqqt1/assignment_loss‚1.1984939891512372 2loss/SRqqt2/assignment_loss‚1.5826614113717243 trainer/global_step‚11299 .loss/FBt3/detection_loss‚1.486024683713913 0loss/FBt3/assignment_loss‚0.3477981711504981 /loss/SRqqt2/detection_loss‚2.53644584787303 .loss/FRt2/detection_loss‚3.116791389309443f<œƒ[)Õ &loss/total_loss‚45.939079851983124 0loss/FRt2/assignment_loss‚3.3550243148436913 /loss/FRt3/assignment_loss‚2.881617928450962 .loss/FBt2/assignment_loss‚0.11094804746308 .loss/FRt3/detection_loss‚2.178065041044973 /loss/FRt4/detection_loss‚1.8368129450629682 1loss/SRqqt1/detection_loss‚1.8180107030333306 1loss/SRqqt3/detection_loss‚1.5262792551746736 .loss/FRt1/assignment_loss‚3.20776914358139 0loss/FRt4/assignment_loss‚1.7257321400845305 .loss/FBt1/detection_loss‚5.757795989513397 % _timestamp‚1.7884255233550737e+09 _runtime‚9023.092534764 2loss/SRqqt1/assignment_loss‚1.1984939891512372 2loss/SRqqt2/assignment_loss‚1.5826614113717243 trainer/global_step‚11299 .loss/FBt3/detection_loss‚1.486024683713913 0loss/FBt3/assignment_loss‚0.3477981711504981 /loss/SRqqt2/detection_loss‚2.53644584787303 .loss/FRt2/detection_loss‚3.116791389309443 1loss/SRqqt4/detection_loss‚1.5390173014180333 /loss/FBt2/detection_loss‚2.1271625638008116 /loss/FBt4/detection_loss‚0.7755953838738302 1loss/FBt4/assignment_loss‚0.06252646587748029 2loss/SRqqt3/assignment_loss‚1.3909269191324711 2loss/SRqqt4/assignment_loss‚0.9335651821548155 /loss/FBt1/assignment_loss‚2.309713721740991 epoch‚9 /loss/FRt1/detection_loss‚2.1343013131618505 _step‚930¢#vQ’)‚(:‡ ºêäÔ„— ~ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[@§ws‘)‚(:è ºêäÔÌ׸€ network.recv‚ 3.267843e+06 disk.sda1.in‚0& disk.nvme0n1.out‚38723.71337890625( disk.nvme0c0n1.out‚38723.71337890625 disk.rbd1.out‚ 4.599609375 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 470.96484375 disk.drbd2266.in‚0 network.sent‚ 2.7481699e+07 disk.nvme0n1.in‚4.6875 disk.rbd0.out‚0 cpu‚25.350951721251217) disk./.usagePercent‚17.55238651452255 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 470.96484375# disk.dm-0.out‚38723.71337890625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 470.96484375$ memory_percent‚94.19965744018555& proc.memory.availableMB‚ 950.328125" proc.memory.rssMB‚ 2772.3984375 proc.cpu.threads‚17$ disk./.usageGB‚261.5432434082031 disk.dm-0.in‚4.6875 disk.rbd1.in‚0* proc.memory.percent‚16.921377182006836 disk.sda1.out‚0 disk.drbd1716.out‚0(Aõƒœ’)‚(:‘ ÉêäÔ‡‘¼~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚155.05' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.010000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ…†Üu“)‚(:ê ÉêäÔ«ýÝ€ disk.rbd1.out‚ 4.599609375 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚38781.63134765625 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875$ memory_percent‚94.19820308685303 disk.sda.out‚ 471.28515625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.273912e+06* disk./.usagePercent‚17.553550836788272 network.sent‚ 2.7590206e+07$ disk./.usageGB‚261.5605926513672 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375( proc.memory.availableMB‚ 950.56640625 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 471.28515625 cpu‚25.35047993179272 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 471.28515625 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17& disk.nvme0n1.out‚38781.63134765625( disk.nvme0c0n1.out‚38781.63134765625 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0Ìß2”)- _wandbruntime‚9058  _runtime‚9058“-Ü]È•) & lr-AdamW/pg1‚1.922923383172708e-06 & lr-AdamW/pg2‚1.922923383172708e-06 trainer/global_step‚11349 % _timestamp‚1.7884255590804465e+09 _runtime‚9058.818001732 _step‚931òD\Í–)Ç & lr-AdamW/pg1‚1.922923383172708e-06 & lr-AdamW/pg2‚1.922923383172708e-06 trainer/global_step‚11349 % _timestamp‚1.7884255590804465e+09 _runtime‚9058.818001732 _step‚931£s»•2—)- _wandbruntime‚9058  _runtime‚9058H–†1#˜)âO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54304494 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚11349 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214385923 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 379582083 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150705201 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721232068694528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721232044754432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9476344892458 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9058.818934994 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96400268 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214385923 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892839424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9975901360128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328814 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454702 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150704762 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165216198 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54304931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚731207946114560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9476344892458 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9058.818934994 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 161526784 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96400268 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214385923 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9511363099136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚670017892839424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9510706736640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 _step‚932 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9975901360128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚652664817328814 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9475690454702 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚660506511737344 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150704762 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96400270 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165216198 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165216198 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670017218473984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605¤!D8ù2š)- _wandbruntime‚9059  _runtime‚90594ãádñ&›)ëM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270651073024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9512182540288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182540288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150717817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9477161327630 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214384339 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚933 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379614553 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404403 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150717372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54308936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048Ÿ«‹ö&œ)ðM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150717372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721293719432192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54308936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731270651073024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚96408436 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165230216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9512182540288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚652720537894338 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379634619 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚643244048044032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9512182540288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚445 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670075065319424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 165230216 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150717817 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9477161327630 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379634619 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214384339 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9476506889666 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165230214 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 _step‚933 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9511526164480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚443 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚660562882779136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 379614553 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚652721209371662 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚660562864776192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9511526164480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚660562864776192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721293695492096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚643244031004672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379614553 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚670074390940672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214404403 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808¥Ú…Rã2)- _wandbruntime‚9059  _runtime‚9059Lm¥%Áž)Ô 1loss/FBt1/assignment_loss‚0.23478427180089056 _runtime‚9059.492772835 /loss/FRt3/assignment_loss‚2.112862573862076 0loss/FRt4/assignment_loss‚1.4361923146755138 1loss/FBt2/assignment_loss‚0.05478656433151059 1loss/FBt4/assignment_loss‚0.12508044685169742 /loss/FRt2/detection_loss‚3.2433887767791747 .loss/FBt1/detection_loss‚2.053998923301697 epoch‚9 1loss/SRqqt1/assignment_loss‚1.318117843029347 2loss/SRqqt2/assignment_loss‚1.9879407500848174 2loss/SRqqt3/assignment_loss‚1.0572539884668704 .loss/FBt3/detection_loss‚2.919701357682546 _step‚934 1loss/SRqqt4/assignment_loss‚1.017327706468528 .loss/FRt1/detection_loss‚2.093110979155258 1loss/SRqqt1/detection_loss‚1.8389108231727114 1loss/SRqqt2/detection_loss‚2.5764257293194532 1loss/SRqqt4/detection_loss‚1.0523150876831653 /loss/FBt4/detection_loss‚0.5018005635428392 %loss/total_loss‚40.56561215531773 /loss/FRt2/assignment_loss‚3.222488007545471 .loss/FRt4/detection_loss‚1.933357893786532 .loss/FBt2/detection_loss‚2.791587209701538 0loss/FBt3/assignment_loss‚0.1285479784113098 1loss/SRqqt3/detection_loss‚1.5026499779178546 trainer/globóf`™al_step‚11349 % _timestamp‚1.7884255597553446e+09 0loss/FRt1/assignment_loss‚3.2319777265742973 /loss/FRt3/detection_loss‚2.1310046611726285Ï^@‰_Ÿ)Ù 1loss/SRqqt4/assignment_loss‚1.017327706468528 .loss/FRt1/detection_loss‚2.093110979155258 1loss/SRqqt1/detection_loss‚1.8389108231727114 1loss/SRqqt2/detection_loss‚2.5764257293194532 1loss/SRqqt4/detection_loss‚1.0523150876831653 /loss/FBt4/detection_loss‚0.5018005635428392 %loss/total_loss‚40.56561215531773 /loss/FRt2/assignment_loss‚3.222488007545471 .loss/FRt4/detection_loss‚1.933357893786532 .loss/FBt2/detection_loss‚2.791587209701538 0loss/FBt3/assignment_loss‚0.1285479784113098 1loss/SRqqt3/detection_loss‚1.5026499779178546 trainer/global_step‚11349 % _timestamp‚1.7884255597553446e+09 0loss/FRt1/assignment_loss‚3.2319777265742973 /loss/FRt3/detection_loss‚2.1310046611726285 1loss/FBt1/assignment_loss‚0.23478427180089056 _runtime‚9059.492772835 /loss/FRt3/assignment_loss‚2.112862573862076 0loss/FRt4/assignment_loss‚1.4361923146755138 1loss/FBt2/assignment_loss‚0.05478656433151059 1loss/FBt4/assignment_loss‚0.12508044685169742 /loss/FRt2/detection_loss‚3.2433887767791747 .loss/FBt1/detection_loss‚2.053998923301697 epoch‚9 1loss/SRqqt1/assignment_loss‚1.318117843029347 2loss/SRqqt2/assignment_loss‚1.9879407500848174 2loss/SRqqt3/assignment_loss‚1.0572539884668704 .loss/FBt3/detection_loss‚2.919701357682546 _step‚934¦9P¦˜’ )‚(:‡ ØêäÔ¤Û¢~ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ŸßÓo¡)‚(:ä ØêäÔ’’æ€ network.sent‚ 2.7596644e+07 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚ 126.77734375$ memory_percent‚94.20294761657715" proc.memory.rssMB‚ 2772.3984375 disk.rbd1.in‚0* proc.memory.percent‚16.921377182006836 cpu‚25.34995599698426 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0$ disk./.usageGB‚261.5629539489746 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd2266.out‚0& disk.nvme0n1.out‚38848.56591796875 disk.rbd1.out‚ 4.599609375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 471.8671875 disk.dm-0.in‚4.6875# disk.dm-0.out‚38848.56591796875 disk.dm-1.out‚ 471.8671875 disk.sda.out‚ 471.8671875 disk.sda1.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚38848.56591796875) disk./.usagePercent‚17.55370930545197 disk.drbd2265.in‚0 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 network.recv‚ 3.27894e+06' proc.memory.availableMB‚ 949.7890625 proc.cpu.threads‚17j>$¢)‚(:’ çêäÔâÀ”~ gpu.0.gpu‚31.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚187.295' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.458999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö¢gµt£)‚(:é çêäÔ•±·€ disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 network.sent‚ 2.7704745e+07$ disk./.usageGB‚261.5723457336426 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375 disk.sda1.in‚0 disk.drbd1716.out‚0& disk.nvme0n1.out‚38895.76123046875( disk.nvme0c0n1.out‚38895.76123046875 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 472.23828125 disk.drbd1716.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2772.3984375# disk.dm-0.out‚38895.76123046875 disk.dm-1.out‚ 472.23828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.283965e+06 disk.dm-0.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0' proc.memory.availableMB‚ 950.2578125 disk.sda3.out‚ 472.23828125$ memory_percent‚94.20008659362793* proc.memory.percent‚16.921377182006836) disk./.usagePercent‚17.55433959600127 disk.drbd2266.out‚0 disk.sda2.in‚0 cpu‚25.349436276867426 proc.cpu.threads‚17 disk.nvme0n1.in‚4.6875)X•ð’¤)‚(:‡ öêäÔæ…—~ gpu.0.gpu‚54.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.623' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó“Qp¥)‚(:å öêäÔصÀ disk.sda.in‚ 126.77734375 disk.dm-1.out‚ 472.97265625 disk.rbd0.in‚0 disk.sda3.out‚ 472.97265625$ memory_percent‚94.20242309570312* proc.memory.percent‚16.921377182006836 cpu‚25.348943035774568 disk.dm-0.in‚4.6875 disk.sda.out‚ 472.97265625 disk.nvme0n1.in‚4.6875# disk.dm-0.out‚38932.41455078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7710736e+07) disk./.usagePercent‚17.55071401730136 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2772.3984375 disk.sda1.in‚0& disk.nvme0n1.out‚38932.41455078125 disk.sda3.in‚ 126.77734375 disk.drbd2265.out‚0$ disk./.usageGB‚261.5183219909668 disk.dm-1.in‚ 126.77734375 disk.rbd0.out‚0 disk.sda2.out‚0# proc.memory.availableMB‚949.875 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚38932.41455078125 disk.sda2.in‚0 network.recv‚ 3.288269e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375wΣ;2¦)- _wandbruntime‚9095  _runtime‚9095kP5ɧ)à $ _timestamp‚1.788425595501767e+09 _runtime‚9095.239322284 _step‚935 ' lr-AdamW/pg1‚1.6730072606064133e-06 ' lr-AdamW/pg2‚1.6730072606064133e-06 trainer/global_step‚11399‡hëoΨ)È $ _timestamp‚1.788425595501767e+09 _runtime‚9095.239322284 _step‚935 ' lr-AdamW/pg1‚1.6730072606064133e-06 ' lr-AdamW/pg2‚1.6730072606064133e-06 trainer/global_step‚11399§’b.€2©)- _wandbruntime‚9095  _runtime‚9095gFo÷è'ª)âO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚672876530587648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165917000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54532636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165916998 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96808670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54533092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734333443502592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673338 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215311975 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚663324181682688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10019027586048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10018861864960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚9095.240249685 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚672875856222208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151341304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692542464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794110 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151341762 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552348904960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚724314581637632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734333633163776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655450859196222åWÂí'«)çO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165916998 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10019027586048 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10018861864960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 _runtime‚9095.240249685 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚672875856222208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151341304 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9551692542464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚663324163679744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9516525794110 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151341762 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9552348904960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9517180231866 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚724314581637632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚734333633163776 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 trainer/global_step‚11399 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚672875856222208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9551692542464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚645934350441472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚663324181682688 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚655450859196222 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚672876530587648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165917000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54532636 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 165916998 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96808670 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚96808668 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54533092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9552348904960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚734333443502592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655451530673338 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381208935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 215311975 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165917000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381228975 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚724314605577728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215291937 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚663324181682688 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430¨û…_2¬)- _wandbruntime‚9095  _runtime‚9095Œ¾M!‚­)îM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517996667038 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151354014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54537178 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚9095.925977359 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396135816704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatòªvrsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703067648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663380534721536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54537657 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553168346112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10019879501312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚481 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511970304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165931014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844`ú×óù&®)óM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚645989254572032 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381261471 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165931014 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381241405 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 198036480 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9517342229074 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 trainer/global_step‚11399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9517996667038 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 165931014 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215310391 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚645989237532672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚672933703067648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚479 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚655506579761746 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚663380516718592 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚724376232375296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151354014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54537178 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚655507251239070 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 _runtime‚9095.925977359 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚734396135816704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚724376256315392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381241405 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚672933703067648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚663380534721536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚663380516718592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54537657 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9553168346112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215330455 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381261471 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10019879501312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174096384 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 165931016 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚481 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9552511970304 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚96816838 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9552511970304©zŒ2¯)- _wandbruntime‚9095  _runtime‚9095Vt{`°)Ú 2loss/SRqqt2/assignment_loss‚1.7539833029431682 epoch‚9 /loss/FRt2/assignment_loss‚3.738033008926055 1loss/SRqqt3/assignment_loss‚1.135271932222639 /loss/FRt4/detection_loss‚1.7440407831893592 1loss/SRqqt3/detection_loss‚1.7091924248746149 /loss/FBt2/detection_loss‚2.8107411935925484 /loss/FBt4/detection_loss‚0.7466034280927214 &loss/total_loss‚40.542898954599124 2loss/SRqqt4/assignment_loss‚0.8824136916762935 1loss/FBt2/assignment_loss‚0.18822060995890766 .loss/FRt1/detection_loss‚2.131877390628165 /loss/FRt2/detection_loss‚3.0016206029583428 /loss/FRt3/detection_loss‚2.1039679271682186 1loss/FBt3/assignment_loss‚0.04661140454700217 2loss/FBt4/assignment_loss‚0.059256598910109884 0loss/FRt4/assignment_loss‚1.4281489946803108 1loss/SRqqt2/detection_loss‚2.3796275690678628 /loss/FBt3/detection_loss‚2.7885062217712404 trainer/global_step‚11399 % _timestamp‚1.7884255961898942e+09 _runtime‚9095.927323189 2loss/SRqqt1/assignment_loss‚1.1989048180963706 0loss/FBt1/assignment_loss‚0.1390298910232054 1loss/SRqqt1/detection_loss‚1.7262406514750586 _step‚938 0loss/FRt1/assignment_loss‚2.8575965204137437 0loss/SRqqt4/detection_loss‚0.927752930785601 /loss/FBt1/detection_loss‚2.2702125774489508 /loss/FRt3/assignment_loss‚2.775044480148627¨×e±)ß 2loss/SRqqt4/assignment_loss‚0.8824136916762935 1loss/FBt2/assignment_loss‚0.18822060995890766 .loss/FRt1/detection_loss‚2.131877390628165 /loss/FRt2/detection_loss‚3.0016206029583428 /loss/FRt3/detection_loss‚2.1039679271682186 1loss/FBt3/assignment_loss‚0.04661140454700217 2loss/FBt4/assignment_loss‚0.059256598910109884 0loss/FRt4/assignment_loss‚1.4281489946803108 1loss/SRqqt2/detection_loss‚2.3796275690678628 /loss/FBt3/detection_loss‚2.7885062217712404 trainer/global_step‚11399 % _timestamp‚1.7884255961898942e+09 _runtime‚9095.927323189 2loss/SRqqt1/assignment_loss‚1.1989048180963706 0loss/FBt1/assignment_loss‚0.1390298910232054 1loss/SRqqt1/detection_loss‚1.7262406514750586 _step‚938 0loss/FRt1/assignment_loss‚2.8575965204137437 0loss/SRqqt4/detection_loss‚0.927752930785601 /loss/FBt1/detection_loss‚2.2702125774489508 /loss/FRt3/assignment_loss‚2.775044480148627 2loss/SRqqt2/assignment_loss‚1.7539833029431682 epoch‚9 /loss/FRt2/assignment_loss‚3.738033008926055 1loss/SRqqt3/assignment_loss‚1.135271932222639 /loss/FRt4/detection_loss‚1.7440407831893592 1loss/SRqqt3/detection_loss‚1.7091924248746149 /loss/FBt2/detection_loss‚2.8107411935925484 /loss/FBt4/detection_loss‚0.7466034280927214 &loss/total_loss‚40.542898954599124ªC¼Áwœ²)‚(:‘ …ëäÔ˜§~ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.372' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚34.27440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[`2çs³)‚(:è …ëäÔäƒË€ disk.drbd2265.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 473.30859375 disk.drbd2080.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚261.5459632873535 disk.drbd2080.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda1.out‚0 disk.sda3.in‚ 126.77734375 cpu‚25.348424201640327 network.recv‚ 3.292209e+06" disk.dm-0.out‚38988.4130859375 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875$ memory_percent‚94.20263767242432* disk./.usagePercent‚17.552569047894462 disk.dm-0.in‚4.6875% disk.nvme0n1.out‚38988.4130859375 disk.sda2.out‚0( proc.memory.availableMB‚ 949.83984375 network.sent‚ 2.7716736e+07 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875' disk.nvme0c0n1.out‚38988.4130859375 disk.dm-1.out‚ 473.30859375 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 473.30859375 disk.sda2.in‚0" proc.memory.rssMB‚ 2772.3984375 disk.drbd2265.out‚0* proc.memory.percent‚16.921377182006836Ò*”´)‚(:… ”ëäÔ™‰Ë~ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!÷Xuµ)‚(:ê ”ëäÔ™ºƒ* proc.memory.percent‚16.921377182006836 network.recv‚ 3.298863e+06 disk.sda3.out‚ 473.91015625 cpu‚25.347932224011252 disk.drbd1716.in‚0 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0 disk.drbd2266.out‚0' proc.memory.availableMB‚ 949.8359375* disk./.usagePercent‚17.552456148571864 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0$ memory_percent‚94.20266151428223" proc.memory.rssMB‚ 2772.3984375 disk.sda.out‚ 473.91015625 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375# disk.dm-0.out‚39052.82373046875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.out‚0$ disk./.usageGB‚261.5442810058594 disk.rbd1.out‚ 4.599609375 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.7825642e+07 disk.rbd1.in‚0( disk.nvme0c0n1.out‚39052.82373046875 disk.drbd2265.out‚0& disk.nvme0n1.out‚39052.82373046875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 473.91015625<Ó¯Ç2¶)- _wandbruntime‚9131  _runtime‚9131䉵|Ê·)Ä ' lr-AdamW/pg2‚1.4403195038443872e-06 trainer/global_step‚11449 % _timestamp‚1.7884256322242007e+09 _runtime‚9131.961723037 _step‚939 ' lr-AdamW/pg1‚1.4403195038443872e-06—VËÛϸ)É trainer/global_step‚11449 % _timestamp‚1.7884256322242007e+09 _runtime‚9131.961723037 _step‚939 ' lr-AdamW/pg1‚1.4403195038443872e-06 ' lr-AdamW/pg2‚1.4403195038443872e-06«Ù"|2¹)- _wandbruntime‚9131  _runtime‚9131Ý‘ÉCé'º)ãO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚675735168335872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9592678348288 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚430 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚11449 _step‚940 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54761664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚666141833625088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666141815622144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9593334710784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54762094 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216238027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557361133518 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166617800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10061869656064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658237572540746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063630 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10062029085696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151978732 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737458988176896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166617798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151979164 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168335872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217989 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚675734493970432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \œbï³!HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚666141815622144 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9593334710784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54762094 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216238027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557361133518 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9592678348288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚727397142460928 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166617800 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166617800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10061869656064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚658237572540746 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚658236901063630 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10062029085696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166617798 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151978732 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚666141815622144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 382855827 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737458988176896 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚648679539930112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 166617798 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151979164 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚675735168335872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216217989 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666198186663936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9593497776128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216236443 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚941 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256507 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666437120 _runtime‚9132.652579773 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521514785280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340815872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166631816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236443 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166631816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151991868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10062898651648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151991440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952íný¾÷&¾)ñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚666198186663936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9593497776128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216236443 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382888323 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚941 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216256507 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675791666437120 _runtime‚9132.652579773 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚737521514785280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675792340815872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚666198168660992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382868257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216256507 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166631816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚666198168660992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97225236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9594154151936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216236443 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚648734444060672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚727458793198592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166631816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151991868 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10062898651648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166631814 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151991440 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382888323 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚727458769258496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚648734461100032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382868257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54766630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54766204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675792340815872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558832006446 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚428 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9594154151936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0­A°ß2¿)- _wandbruntime‚9132  _runtime‚9132+>™p]À)× 0loss/FRt2/assignment_loss‚3.2279439588387806 0loss/FRt4/assignment_loss‚1.0250945780012342 0loss/FBt3/assignment_loss‚0.2172827762842644 epoch‚9 0loss/FBt1/assignment_loss‚0.1446439943421808 .loss/FRt1/detection_loss‚2.230137300880059 1loss/SRqqt1/detection_loss‚1.9903714168816804 /loss/FBt4/detection_loss‚0.5894175396533683 -loss/FRt3/detection_loss‚2.34490888998952 1loss/SRqqt3/detection_loss‚1.9075875447555022 1loss/SRqqt1/assignment_loss‚1.423003511922434 2loss/SRqqt4/assignment_loss‚0.9474469318554022 /loss/FBt1/detection_loss‚2.2886499017477036 .loss/FBt3/detection_loss‚1.748685047030449 % _timestamp‚1.7884256329161963e+09 0loss/FRt3/assignment_loss‚2.8095951080322266 2loss/SRqqt3/assignment_loss‚1.3433182347565888 1loss/FBt4/assignment_loss‚0.09749647990805711 2loss/SRqqt2/assignment_loss‚1.8278162674261973 1loss/FBt2/assignment_loss‚0.32594138038176523 /loss/FRt2/detection_loss‚2.9384540021419525 _step‚942 /loss/FRt4/detection_loss‚1.9153315194778973 1loss/SRqqt2/detection_loss‚2.1693267004612165 1loss/SRqqt4/detection_loss‚0.8029188966189605 trainer/global_step‚11449 /loss/FBt2/detection_loss‚2.3250482281049094 %loss/total_loss‚39.77088507560187 _runtime‚9132.653936403 0loss/FRt1/assignment_loss‚3.1304648661095165QÉÁbÁ)Ü 0loss/FRt1/assignment_loss‚3.1304648661095165 0loss/FRt2/assignment_loss‚3.2279439588387806 0loss/FRt4/assignment_loss‚1.0250945780012342 0loss/FBt3/assignment_loss‚0.2172827762842644 epoch‚9 0loss/FBt1/assignment_loss‚0.1446439943421808 .loss/FRt1/detection_loss‚2.230137300880059 1loss/SRqqt1/detection_loss‚1.9903714168816804 /loss/FBt4/detection_loss‚0.5894175396533683 -loss/FRt3/detection_loss‚2.34490888998952 1loss/SRqqt3/detection_loss‚1.9075875447555022 1loss/SRqqt1/assignment_loss‚1.423003511922434 2loss/SRqqt4/assignment_loss‚0.9474469318554022 /loss/FBt1/detection_loss‚2.2886499017477036 .loss/FBt3/detection_loss‚1.748685047030449 % _timestamp‚1.7884256329161963e+09 0loss/FRt3/assignment_loss‚2.8095951080322266 2loss/SRqqt3/assignment_loss‚1.3433182347565888 1loss/FBt4/assignment_loss‚0.09749647990805711 2loss/SRqqt2/assignment_loss‚1.8278162674261973 1loss/FBt2/assignment_loss‚0.32594138038176523 /loss/FRt2/detection_loss‚2.9384540021419525 _step‚942 /loss/FRt4/detection_loss‚1.9153315194778973 1loss/SRqqt2/detection_loss‚2.1693267004612165 1loss/SRqqt4/detection_loss‚0.8029188966189605 trainer/global_step‚11449 /loss/FBt2/detection_loss‚2.3250482281049094 %loss/total_loss‚39.77088507560187 _runtime‚9132.653936403® 5iÂ)‚(:„ £ëäÔÕÙÑ gpu.0.gpu‚47.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.15' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.03 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚083sÃ)‚(:ä £ëäÔꌺ network.recv‚ 3.302505e+06( disk./.usagePercent‚17.5537177537006& disk.nvme0n1.out‚39097.26904296875 disk.dm-1.out‚ 474.51953125 disk.drbd2266.out‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.7830926e+07 disk.nvme0c0n1.in‚4.6875 disk.rbd1.out‚ 4.599609375$ memory_percent‚94.20280456542969 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 474.51953125 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375$ proc.memory.availableMB‚949.8125 proc.cpu.threads‚17 disk.sda2.out‚0# disk.dm-0.out‚39097.26904296875 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sdÒ°œ üa.out‚ 474.51953125 cpu‚25.34744195899044$ disk./.usageGB‚261.5630798339844 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚39097.26904296875 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875* proc.memory.percent‚16.921377182006836)3ž’Ä)‚(:‡ ²ëäÔ±ø¥~ gpu.0.gpu‚32.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.466' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¼zpÅ)‚(:å ²ëäÔ©–§ disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 475.1328125 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.rbd1.out‚ 4.599609375# disk.dm-0.out‚39162.09326171875 proc.cpu.threads‚17 network.recv‚ 3.308544e+06* disk./.usagePercent‚17.555052832991663 disk.sda2.in‚0 disk.drbd2080.out‚0$ memory_percent‚94.20299530029297* proc.memory.percent‚16.921377182006836$ disk./.usageGB‚261.5829734802246 disk.nvme0n1.in‚4.6875 disk.dm-1.out‚ 475.1328125 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 949.78125" proc.memory.rssMB‚ 2772.3984375 network.sent‚ 2.7939554e+07 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.346983890832888 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚39162.09326171875 disk.sda3.in‚ 126.77734375 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 475.1328125 disk.sda1.out‚0& disk.nvme0n1.out‚39162.09326171875ɾãU’Æ)‚(:‡ ÁëäÔ…µ“~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚167.633' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.5266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯W=hlÇ)‚(:á ÁëäÔþ”& disk.nvme0n1.out‚39214.13623046875 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚39214.13623046875$ memory_percent‚94.20151710510254" proc.memory.rssMB‚ 2772.3984375* proc.memory.percent‚16.921377182006836 disk.rbd1.in‚0 cpu‚25.346472877490893 proc.cpu.threads‚17* disk./.usagePercent‚17.555793206780685$ disk./.usageGB‚261.5940055847168 disk.drbd2080.out‚0 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 475.59375 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0' proc.memory.availableMB‚ 950.0234375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 475.59375 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚4.6875 disk.drbd1716.out‚0 disk.rbd0.in‚0 network.sent‚ 2.7945436e+07# disk.dm-0.out‚39214.13623046875 disk.sda3.out‚ 475.59375 disk.drbd2265.out‚0 network.recv‚ 3.312518e+06 disk.sda.in‚ 126.77734375 o“ÿ2È)- _wandbruntime‚9168  _runtime‚9168$û¸ÒÊÉ)Ä ' lr-AdamW/pg1‚1.2249086046663712e-06 ' lr-AdamW/pg2‚1.2249086046663712e-06 trainer/global_step‚11499 % _timestamp‚1.7884256691734126e+09 _runtime‚9168.910986117 _step‚943Ù4-ÏÊ)É ' lr-AdamW/pg1‚1.2249086046663712e-06 ' lr-AdamW/pg2‚1.2249086046663712e-06 trainer/global_step‚11499 % _timestamp‚1.7884256691734126e+09 _runtime‚9168.910986117 _step‚943¯ µ2Ë)- _wandbruntime‚9168  _runtime‚9168O%é'Ì)ãO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664154112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚11499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788425669174096e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10104822932992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152615888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668959485567488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217164079 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54990888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54990420 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10104976071168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9598850910682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496ž î'Í)èO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚668959467564544 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678593131718656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10104822932992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9598196472926 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9634320516608 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152615888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚651424746458112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚668959485567488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217164079 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚678593806084096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661023614408154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217164079 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9634320516608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54990888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54990420 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678593131718656 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚668959467564544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10104976071168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9598850910682 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217144041 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661022942931038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9633664154112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384482679 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚730479655404032 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 trainer/global_step‚11499 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 177078272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚651424763497472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167318600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚668959485567488 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 167318598 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚730479679344128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 $ _timestamp‚1.788425669174096e+09°3ŠÁx2Î)- _wandbruntime‚9169  _runtime‚9169¤rg¨£ Ï)îM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669015838606336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678650978564096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54994940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217182559 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907890 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167332616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669015838606336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152629040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9635139957760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740646985431040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9599667345854 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651479667628032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162495 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647172981760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661079334973886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384495109 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54995402 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ à;&Q9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167332616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10105842899968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152628576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10105679289344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚945Xl­Qù&Ð)óM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669015838606336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9634483581952 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚678650978564096 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54994940 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚651479650588672 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217182559 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9599012907890 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167332616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669015838606336 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669015820603392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 187550720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152629040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚730541306141696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9635139957760 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167332614 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚97633636 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740646985431040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9599667345854 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚730541330081792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384515175 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚651479667628032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 163610624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217162495 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740647172981760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661079334973886 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 384495109 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167332614 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54995402 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167332616 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10105842899968 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152628576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10105679289344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚945 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0±«úó[2Ñ)- _wandbruntime‚9169  _runtime‚9169{¹¹ËZÒ)Ô epoch‚9 0loss/SRqqt4/assignment_loss‚0.94429328988278 1loss/FBt4/assignment_loss‚0.15740120682365466 /loss/FRt1/detection_loss‚2.4474940961453973 /loss/FRt3/detection_loss‚2.6848344217985867 _runtime‚9169.601084747 0loss/FRt4/assignment_loss‚1.5780942797660829 2loss/SRqqt3/assignment_loss‚1.3829137237044051 1loss/SRqqt3/detection_loss‚1.6270953633955547 /loss/FRt1/assignment_loss‚3.058624481282583 0loss/SRqqt2/detection_loss‚2.410836338996887 _step‚946 2loss/SRqqt1/assignment_loss‚1.3410257204267821 1loss/FBt3/assignment_loss‚0.23223546007647636 /loss/FRt4/detection_loss‚1.9265739717003372 .loss/FBt1/detection_loss‚2.532452302319663 0loss/FBt1/assignment_loss‚0.0912975248640058 0loss/FBt2/assignment_loss‚0.3478555461280657 .loss/FRt2/detection_loss‚2.918791389465332 %loss/total_loss‚41.09920678252091 % _timestamp‚1.7884256698635733e+09 /loss/FRt2/assignment_loss‚3.054330535729726 2loss/SRqqt2/assignment_loss‚1.5555105805397034 trainer/global_step‚11499 0loss/FRt3/assignment_loss‚2.7605999559164047 1loss/SRqqt1/detection_loss‚1.6895068072881856 1loss/SRqqt4/detection_loss‚0.7771529378087232 /loss/FBt2/detection_loss‚2.6475043947046455 /loss/FBt3/detection_loss‚2.4405824144681296 .loss/FBt4/detection_loss‚0.492200039288796¸áÛ_Ó)Ù epoch‚9 0loss/SRqqt4/assignment_loss‚0.94429328988278 1loss/FBt4/assignment_loss‚0.15740120682365466 /loss/FRt1/detection_loss‚2.4474940961453973 /loss/FRt3/detection_loss‚2.6848344217985867 _runtime‚9169.601084747 0loss/FRt4/assignment_loss‚1.5780942797660829 2loss/SRqqt3/assignment_loss‚1.3829137237044051 1loss/SRqqt3/detection_loss‚1.6270953633955547 /loss/FRt1/assignment_loss‚3.058624481282583 0loss/SRqqt2/detection_loss‚2.410836338996887 _step‚946 2loss/SRqqt1/assignment_loss‚1.3410257204267821 1loss/FBt3/assignment_loss‚0.23223546007647636 /loss/FRt4/detection_loss‚1.9265739717003372 .loss/FBt1/detection_loss‚2.532452302319663 0loss/FBt1/assignment_loss‚0.0912975248640058 0loss/FBt2/assignment_loss‚0.3478555461280657 .loss/FRt2/detection_loss‚2.918791389465332 %loss/total_loss‚41.09920678252091 % _timestamp‚1.7884256698635733e+09 /loss/FRt2/assignment_loss‚3.054330535729726 2loss/SRqqt2/assignment_loss‚1.5555105805397034 trainer/global_step‚11499 0loss/FRt3/assignment_loss‚2.7605999559164047 1loss/SRqqt1/detection_loss‚1.6895068072881856 1loss/SRqqt4/detection_loss‚0.7771529378087232 /loss/FBt2/detection_loss‚2.6475043947046455 /loss/FBt3/detection_loss‚2.4405824144681296 .loss/FBt4/detection_loss‚0.492200039288796²v9˜Ô)‚(:… ÐëäÔ—‚© gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.898 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0䃞ZmÕ)‚(:â ÐëäÔÜù€" proc.memory.rssMB‚ 2772.3984375 disk.rbd0.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.in‚0 network.sent‚ 2.7951771e+07 disk.rbd1.out‚ 4.599609375$ memory_percent‚94.20318603515625' disk.nvme0c0n1.out‚39272.5498046875 disk.sda3.out‚ 476.18359375 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.sda.out‚ 476.18359375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚39272.5498046875" proc.memory.availableMB‚949.75 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2080.out‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.921377182006836 proc.cpu.threads‚17$ disk./.usageGB‚261.6046562194824 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.drbd2080.in‚0 cpu‚25.345995804552455 disk.nvme0c0n1.in‚4.6875 disk.dm-1.out‚ 476.18359375% disk.nvme0n1.out‚39272.5498046875 network.recv‚ 3.317454e+06* disk./.usagePercent‚17.556507979816285 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875Iô:’Ö)‚(:‡ ßëäÔߪû} gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.2454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ùóAa×)‚(:Ö ßëäÔÆŒˆ€ disk.sda3.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.out‚ 476.53125 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875 disk.dm-0.in‚4.6875 disk.dm-1.out‚ 476.53125 cpu‚25.34554166402728 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375$ memory_percent‚94.20318603515625" proc.memory.availableMB‚949.75 network.sent‚ 2.8059667e+07 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2772.3984375 disk.drbd2265.out‚0! disk.dm-0.out‚39325.244140625 network.recv‚ 3.323125e+06$ disk.nvme0n1.out‚39325.244140625 disk.drbd2080.out‚0 proc.cpu.threads‚17% disk./.usageGB‚261.61335372924805 disk.drbd2266.out‚0 disk.rbd1.out‚ 4.599609375 disk.sda1.out‚0 disk.sda3.out‚ 476.53125 disk.drbd2265.in‚0* proc.memory.percent‚16.921377182006836* disk./.usagePercent‚17.557091676994354 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚39325.244140625 disk.sda2.in‚0?x•42Ø)- _wandbruntime‚9205  _runtime‚9205õW»ÚÊÙ)Ä % _timestamp‚1.7884257059569566e+09 _runtime‚9205.694470245 _step‚947 ' lr-AdamW/pg1‚1.0268194543808886e-06 ' lr-AdamW/pg2‚1.0268194543808886e-06 trainer/global_step‚115492 zÏÚ)É ' lr-AdamW/pg2‚1.0268194543808886e-06 trainer/global_step‚11549 % _timestamp‚1.7884257059569566e+09 _runtime‚9205.694470245 _step‚947 ' lr-AdamW/pg1‚1.0268194543808886e-06³K ~2Û)- _wandbruntime‚9205  _runtime‚9205\/e‡ê'Ü)äO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671777119506944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168019400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚948 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671777119506944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98033868 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681451769466880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639031812334 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654169970025472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153253581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55219250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070093 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649959936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306322432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710335879680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710148315648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090131 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55219711 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671777137509888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275562 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681452443832320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884257059576433e+09 _runtime‚9205.695357219 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10148119652352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10147956028416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÅÙŠ‹Ý)éO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168019398 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884257059576433e+09 _runtime‚9205.695357219 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98033870 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 163623936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9675306322432 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 386109531 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218090131 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10148119652352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10147956028416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚671777119506944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168019400 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218070093 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚663808984798446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 _step‚948 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9674649959936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9639686250090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚671777119506944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚463 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98033868 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681451769466880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 @8DeviceStatsMonitor.oÔkBÃdn_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681451769466880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9639031812334 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654169970025472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153253581 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55219250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218070093 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9674649959936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚461 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9675306322432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 386089491 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚743710335879680 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚743710148315648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚671777137509888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733562192287232 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218090131 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55219711 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚671777137509888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚663809656275562 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚681452443832320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 187564032 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941933568 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98042036 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363970 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168033414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9675469387776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681509616312320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153265945 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10148838857216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9676125763584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743772854752768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153266381 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55223909 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10148987787776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108611 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚9206.386022715 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 386142027 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98042036 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663864705363970 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 148930560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚436 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚671833472545792 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168033414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9675469387776 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 386121961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚681509616312320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153265945 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10148838857216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9676125763584 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚743772854752768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚671833490548736 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153266381 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55223909 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10148987787776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218108611 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 _step‚949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9676125763584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚9206.386022715 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚733623843024896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168033416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 386142027 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚681508941933568 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168033414 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168720198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746835880520704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9716292128256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219016183 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407215104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9715635765760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736644729170432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191319168512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9244.294264037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚746836072279040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788425744556489e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153891458 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168720200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219016183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635765760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407215104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9680521589498 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191151350272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚11599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168720198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚746835880520704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚674594789452288 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚674594771449344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9716292128256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219016183 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚684310407215104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9715635765760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚736644729170432 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10191319168512 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656915159514112 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 _runtime‚9244.294264037 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98442268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚746836072279040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788425744556489e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153891458 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168720200 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219016183 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9715635765760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 168720198 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218996145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚684310407215104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9680521589498 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218996145 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 _step‚952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10191151350272 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387716343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚466 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 trainer/global_step‚11599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¸%jé 2ð)- _wandbruntime‚9245  _runtime‚9245sç·úô&ñ)îM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254060544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153904674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚746898588867584 _step‚953 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684367579681792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9717111569408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55453787 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55454236 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98450438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219034663 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219034663 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748813 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219014599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153904223 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884257452740757e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219014599 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746898413899776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚11599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9716455193600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9680683586706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872ºV]ûíò)óM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚746898588867584 _step‚953 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚684367579681792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9717111569408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387768879 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55453787 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656970063644672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98450436 VI/U” "DeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55454236 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98450438 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168734214 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219034663 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚736706379908096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 151027712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219034663 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168734216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168734216 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387748813 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387748813 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚674651124488192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219014599 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153904223 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 174967808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884257452740757e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚674651142491136 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9716455193600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387768879 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚674651142491136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9717111569408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219014599 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚746898413899776 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚11599 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚736706403848192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9716455193600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9680683586706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚674651124488192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚684368254060544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153904674 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¹QpC2ó)- _wandbruntime‚9245  _runtime‚9245EÄXô)Ò 1loss/SRqqt1/detection_loss‚1.8667245957909562 /loss/FBt2/detection_loss‚2.7934751311937966 trainer/global_step‚11599 .loss/FRt2/assignment_loss‚3.38837983106312 2loss/SRqqt2/assignment_loss‚1.6877811771745865 1loss/SRqqt4/detection_loss‚1.0483096075662668 .loss/FBt1/detection_loss‚2.495800459384918 /loss/FBt4/detection_loss‚0.5305294944838671 1loss/SRqqt1/assignment_loss‚1.339845655531418 1loss/SRqqt3/assignment_loss‚1.399339386047079 0loss/FBt1/assignment_loss‚0.6989984685787931 1loss/FBt2/assignment_loss‚0.10169184242802842 1loss/SRqqt2/detection_loss‚2.3984551292199354 0loss/FRt1/assignment_loss‚3.8996586979821672 -loss/FRt1/detection_loss‚2.24118601860002 1loss/SRqqt3/detection_loss‚1.6193089601225579 _runtime‚ 9245.01343192 _step‚954 0loss/FRt3/assignment_loss‚2.7124878854855248 2loss/SRqqt4/assignment_loss‚0.6810155389256829 1loss/FBt4/assignment_loss‚0.12298215645732204 /loss/FRt4/detection_loss‚1.7848668942308945 .loss/FBt3/detection_loss‚4.176169514656067 .loss/FRt3/detection_loss‚2.438899020137994 %loss/total_loss‚43.62548305715445 epoch‚9 % _timestamp‚1.7884257452759893e+09 0loss/FRt4/assignment_loss‚1.4598688509153284 1loss/FBt3/assignment_loss‚0.15377213054216554 .loss/FRt2/detection_loss‚2.585936610635958Ÿ_„Ñ]õ)× _step‚954 0loss/FRt3/assignment_loss‚2.7124878854855248 2loss/SRqqt4/assignment_loss‚0.6810155389256829 1loss/FBt4/assignment_loss‚0.12298215645732204 /loss/FRt4/detection_loss‚1.7848668942308945 .loss/FBt3/detection_loss‚4.176169514656067 .loss/FRt3/detection_loss‚2.438899020137994 %loss/total_loss‚43.62548305715445 epoch‚9 % _timestamp‚1.7884257452759893e+09 0loss/FRt4/assignment_loss‚1.4598688509153284 1loss/FBt3/assignment_loss‚0.15377213054216554 .loss/FRt2/detection_loss‚2.585936610635958 1loss/SRqqt1/detection_loss‚1.8667245957909562 /loss/FBt2/detection_loss‚2.7934751311937966 trainer/global_step‚11599 .loss/FRt2/assignment_loss‚3.38837983106312 2loss/SRqqt2/assignment_loss‚1.6877811771745865 1loss/SRqqt4/detection_loss‚1.0483096075662668 .loss/FBt1/detection_loss‚2.495800459384918 /loss/FBt4/detection_loss‚0.5305294944838671 1loss/SRqqt1/assignment_loss‚1.339845655531418 1loss/SRqqt3/assignment_loss‚1.399339386047079 0loss/FBt1/assignment_loss‚0.6989984685787931 1loss/FBt2/assignment_loss‚0.10169184242802842 1loss/SRqqt2/detection_loss‚2.3984551292199354 0loss/FRt1/assignment_loss‚3.8996586979821672 -loss/FRt1/detection_loss‚2.24118601860002 1loss/SRqqt3/detection_loss‚1.6193089601225579 _runtime‚ 9245.01343192ºéA€’ö)‚(:‡ ›ìäÔçåµ~ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.964' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/M©°q÷)‚(:æ ›ìäÔ㵌# proc.memory.rssMB‚ 2691.87109375$ disk.nvme0n1.out‚39543.255859375 disk.rbd0.in‚0 disk.sda3.out‚ 479.12109375 disk.drbd2265.in‚0! disk.dm-0.out‚39543.255859375 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚ 4.7802734375 network.recv‚ 3.341594e+06 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚4.6875 cpu‚25.3435983401449% disk./.usageGB‚261.61791229248047 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375) proc.memory.availableMB‚ 1028.92578125 network.sent‚ 2.8185847e+07* disk./.usagePercent‚17.557397605997767 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda.out‚ 479.12109375 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-1.out‚ 479.12109375# memory_percent‚93.7199354171753& disk.nvme0c0n1.out‚39543.255859375 disk.drbd2265.out‚0* proc.memory.percent‚16.429877281188965óÿK’ø)‚(:‡ ªìäÔ½Ù²~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.872' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.1744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D,à©pù)‚(:å ªìäÔÉ’š network.sent‚ 2.8294427e+07) disk./.usagePercent‚17.55883995244568 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 network.recv‚ 3.347804e+06 disk.dm-0.in‚4.6875& disk.nvme0c0n1.out‚39597.849609375# proc.memory.rssMB‚ 2693.20703125 cpu‚25.343123924843226# disk./.usageGB‚261.639404296875 disk.sda.out‚ 479.95703125 disk.nvme0n1.in‚4.6875 disk.sda2.out‚0! disk.dm-0.out‚39597.849609375 disk.rbd1.out‚ 4.84326171875 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 1029.1640625 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.drbd2080.out‚0) proc.memory.percent‚16.43803119659424 disk.drbd2265.in‚0$ memory_percent‚93.71848106384277 disk.sda3.out‚ 479.95703125 disk.dm-1.out‚ 479.95703125$ disk.nvme0n1.out‚39597.849609375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375Ý2ñ¿2ú)- _wandbruntime‚9281  _runtime‚9281 FWÈû) _runtime‚9281.904373971 _step‚955 & lr-AdamW/pg1‚6.827679079861176e-07 & lr-AdamW/pg2‚6.827679079861176e-07 trainer/global_step‚11649 % _timestamp‚1.7884257821668448e+09-öÏèÍü)Ç & lr-AdamW/pg1‚6.827679079861176e-07 & lr-AdamW/pg2‚6.827679079861176e-07 trainer/global_step‚11649 % _timestamp‚1.7884257821668448e+09 _runtime‚9281.904373971 _step‚955»&ó›¥2ý)- _wandbruntime‚9281  _runtime‚9281»¡6ê'þ)äO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961590916608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219942235 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _runtime‚9281.905270162 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169421000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277934080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219922197 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚11649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044963328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219942235 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677412441394688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961761703424 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10234471709696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚659660383081472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9756621571584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55678533 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010378 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169420998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234324862976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚956 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 170786816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9721356928906 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9757277934080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9756621571584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154529653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 146846720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677412423391744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0þ“ Ãÿ)éO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10234471709696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚659660383081472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9756621571584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 169420998 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389363235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55678533 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚669381740010378 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚687169719328768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677412423391744 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169420998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98850670 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10234324862976 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚956 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 170786816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9721356928906 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9757277934080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9756621571584 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389343195 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 154529653 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 146846720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677412423391744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739727289993728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚450 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚659660366042112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677412441394688 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚749961590916608 LFDeviceSt~ט, atsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚687169044963328 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219942235 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 _runtime‚9281.905270162 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 169421000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9757277934080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219922197 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 trainer/global_step‚11649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚687169044963328 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219942235 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚677412441394688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749961761703424¼A7xÖ2€*- _wandbruntime‚9282  _runtime‚9282[ȯô&*îM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169435014 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219940651 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚669437460576110 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097375232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024282953216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235342221824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024087013888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739788916791296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169435014 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659715287212032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219940651 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677468776430592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9721518926114 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚669436789098786 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389375665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10235170222592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884257828655498e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677468776430592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154541972 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¨}sLù&‚*óM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169435014 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219960715 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219940651 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚669437460576110 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9758097375232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚98858836 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚462 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚750024282953216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169435016 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10235342221824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚750024087013888 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚739788916791296 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 169435014 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚659715287212032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389395731 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219940651 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚677468776430592 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9757440999424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚739788940731392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389395731 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9721518926114 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚669436789098786 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚659715270172672 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 389375665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10235170222592 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 % _timestamp‚1.7884257828655498e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677468776430592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154541972 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389375665½5@áœ2ƒ*- _wandbruntime‚9282  _runtime‚9282å}–Z„*Ô 0loss/FRt3/assignment_loss‚3.5439257234334947 0loss/SRqqt3/detection_loss‚1.651356330762307 1loss/SRqqt3/assignment_loss‚1.345433447137475 _runtime‚ 9282.60478156 /loss/FRt1/assignment_loss‚3.612849390820453 0loss/FRt4/assignment_loss‚1.7832799947008173 2loss/SRqqt2/assignment_loss‚2.1607338182868494 2loss/SRqqt4/assignment_loss‚1.3424923812801188 2loss/FBt1/assignment_loss‚0.006429905470992837 0loss/FBt2/assignment_loss‚0.1938035387996706 _step‚958 0loss/FRt2/assignment_loss‚3.0782118290662766 1loss/SRqqt1/assignment_loss‚1.294495449927838 1loss/FBt3/assignment_loss‚0.13782521944293485 .loss/FRt4/detection_loss‚1.529909988904887 1loss/SRqqt2/detection_loss‚2.6153855939065256 0loss/SRqqt4/detection_loss‚1.007976532208197 /loss/FBt4/detection_loss‚0.4738963629365955 &loss/total_loss‚42.758680661165734 epoch‚9 trainer/global_step‚11649 -loss/FRt2/detection_loss‚2.77064673230052 /loss/FBt3/detection_loss‚2.7628353408404758 % _timestamp‚1.7884257828670356e+09 1loss/FBt4/assignment_loss‚0.09910129890776176 .loss/FRt1/detection_loss‚2.246512314990947 /loss/FRt3/detection_loss‚2.3545796029269694 /loss/FBt1/detection_loss‚2.5436320390020097 .loss/FBt2/detection_loss‚2.170434892177582 1loss/SRqqt1/detection_loss‚2.0329329329340355©^i_…*Ù 0loss/FBt2/assignment_loss‚0.1938035387996706 _step‚958 0loss/FRt2/assignment_loss‚3.0782118290662766 1loss/SRqqt1/assignment_loss‚1.294495449927838 1loss/FBt3/assignment_loss‚0.13782521944293485 .loss/FRt4/detection_loss‚1.529909988904887 1loss/SRqqt2/detection_loss‚2.6153855939065256 0loss/SRqqt4/detection_loss‚1.007976532208197 /loss/FBt4/detection_loss‚0.4738963629365955 &loss/total_loss‚42.758680661165734 epoch‚9 trainer/global_step‚11649 -loss/FRt2/detection_loss‚2.77064673230052 /loss/FBt3/detection_loss‚2.7628353408404758 % _timestamp‚1.7884257828670356e+09 1loss/FBt4/assignment_loss‚0.09910129890776176 .loss/FRt1/detection_loss‚2.246512314990947 /loss/FRt3/detection_loss‚2.3545796029269694 /loss/FBt1/detection_loss‚2.5436320390020097 .loss/FBt2/detection_loss‚2.170434892177582 1loss/SRqqt1/detection_loss‚2.0329329329340355 0loss/FRt3/assignment_loss‚3.5439257234334947 0loss/SRqqt3/detection_loss‚1.651356330762307 1loss/SRqqt3/assignment_loss‚1.345433447137475 _runtime‚ 9282.60478156 /loss/FRt1/assignment_loss‚3.612849390820453 0loss/FRt4/assignment_loss‚1.7832799947008173 2loss/SRqqt2/assignment_loss‚2.1607338182868494 2loss/SRqqt4/assignment_loss‚1.3424923812801188 2loss/FBt1/assignment_loss‚0.006429905470992837¾ìµ¥"’†*‚(:‡ ¹ìäÔÕÚé~ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.846' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.7692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Ö,¢c‡*‚(:Ø ¹ìäԜᖀ disk.drbd2080.out‚0 disk.dm-0.in‚4.6875 disk.rbd0.out‚0) disk./.usagePercent‚17.55955805357922 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.43803119659424 disk.nvme0n1.in‚4.6875 disk.dm-0.out‚39643.97265625 disk.dm-1.out‚480.3125 disk.rbd1.out‚ 4.84326171875 cpu‚25.342630672206987 disk.drbd1716.out‚0# disk.nvme0n1.out‚39643.97265625 network.sent‚ 2.8300289e+07$ disk./.usageGB‚261.6501045227051 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.drbd2266.out‚0$ memory_percent‚93.72012615203857 disk.rbd0.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1028.89453125# proc.memory.rssMB‚ 2693.20703125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚4.6875 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda.out‚480.3125 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚480.3125 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375 network.recv‚ 3.352039e+06 disk.sda1.in‚0% disk.nvme0c0n1.out‚39643.97265625ì8ù¡’ˆ*‚(:‡ ÈìäÔÝñ‹~ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.252' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃØ>Ad‰*‚(:Ù ÈìäÔøæÖ€ disk.rbd0.out‚0) proc.memory.availableMB‚ 1028.62890625 disk.rbd1.out‚4.90625# disk.nvme0n1.out‚39695.59765625 disk.drbd2080.out‚0 disk.dm-0.in‚4.6875) proc.memory.percent‚16.43803119659424 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 480.890625# proc.memory.rssMB‚ 2693.20703125 proc.cpu.threads‚17 network.sent‚ 2.8409302e+07 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0$ memory_percent‚93.72174739837646 disk.drbd2266.out‚0 disk.sda.out‚ 480.890625 disk.sda1.in‚0 disk.drbd2265.out‚0 cpu‚25.342157218767507$ disk./.usageGB‚261.6694221496582 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 480.890625 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 network.recv‚ 3.358612e+06% disk.nvme0c0n1.out‚39695.59765625 disk.dm-0.out‚39695.59765625* disk./.usagePercent‚17.560854475732615†çíŠ*‚(:… ×ìäÔÁ˜ú} gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚173.18' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.636 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¬œåc‹*‚(:Ø ×ìäÔÔÈÿ€ disk.dm-0.out‚39740.02734375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 1029.6171875 disk.nvme0n1.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 481.3203125 disk.sda1.in‚0) proc.memory.percent‚16.43803119659424 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd1.out‚4.90625 disk.sda1.out‚0# memory_percent‚93.7157154083252 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd1716.out‚0 cpu‚25.341686274331817 disk.sda3.out‚ 481.3203125 network.sent‚ 2.841456e+07% disk.nvme0c0n1.out‚39740.02734375 disk.dm-0.in‚4.6875 network.recv‚ 3.362063e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda.out‚ 481.3203125 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.20703125) disk./.usagePercent‚17.55124497692737$ disk./.usageGB‚261.5262336730957# disk.nvme0n1.out‚39740.02734375 disk.drbd2080.out‚0v‘Ìÿ2Œ*- _wandbruntime‚9319  _runtime‚9319›ªtÇ*Á & lr-AdamW/pg1‚5.368772117033979e-07 & lr-AdamW/pg2‚5.368772117033979e-07 trainer/global_step‚11699 $ _timestamp‚1.788425819620543e+09 _runtime‚9319.358113616 _step‚959›Ï‚ÌŽ*Æ $ _timestamp‚1.788425819620543e+09 _runtime‚9319.358113616 _step‚959 & lr-AdamW/pg1‚5.368772117033979e-07 & lr-AdamW/pg2‚5.368772117033979e-07 trainer/global_step‚11699¿æUµ2*- _wandbruntime‚9319  _runtime‚93191"Õkw*äO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170121800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55909347 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10277313183232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680230093337088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390990087 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672167781877786 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220868287 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99259070 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220868287 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9798263739904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9798263739904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9761537830558 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170121798 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220848249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690028357076992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55908888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _runtime‚9319.359091571 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155167956 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087116120064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848249 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680230093337088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712rrAjï'‘*éO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99259068 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚753087307878400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662405572570112 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 390970047 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662405589609472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170121800 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220848249 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚680230093337088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170121800 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55909347 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10277313183232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚680230093337088 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390990087 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚672167781877786 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚680230075334144 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220868287 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99259070 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 220868287 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9798263739904 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9798263739904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚459 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9761537830558 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚742809826876928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690027682711552 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 170121798 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220848249 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690028357076992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55908888 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 _runtime‚9319.359091571 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155167956 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚753087116120064 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690084855178240 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170135814 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155180634 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170135814 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9799083181056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99267236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672222830966194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚742871477614592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443518 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662460493740032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944ÄÓ3³U”*óM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚672223502443518 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662460493740032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690085529556992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚680286446375936 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚680286428372992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚690084855178240 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170135816 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170135814 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155180634 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚742871453674496 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170135814 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391022583 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9799083181056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9799083181056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚471 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 trainer/global_step‚11699 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170135816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99267236 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚672222830966194 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 _step‚961 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚690085529556992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚742871477614592Áv‹Ÿ˜2•*- _wandbruntime‚9320  _runtime‚9320¸Á*ÊZ–*Ô 1loss/FBt1/assignment_loss‚0.37693514972208786 .loss/FBt2/detection_loss‚2.769229610761007 _step‚962 $ _timestamp‚1.788425820313731e+09 epoch‚9 trainer/global_step‚11699 0loss/FRt3/assignment_loss‚2.5181640934944154 .loss/FRt1/detection_loss‚2.377264790946529 1loss/SRqqt1/assignment_loss‚1.060406762884841 2loss/SRqqt3/assignment_loss‚1.1270433824864172 .loss/FRt2/detection_loss‚2.773805797100067 1loss/SRqqt3/detection_loss‚1.3842759198612637 .loss/FBt3/detection_loss‚2.229116380214691 /loss/FRt1/assignment_loss‚3.469596754936945 1loss/FBt4/assignment_loss‚0.09345068854893213 /loss/FRt3/detection_loss‚2.1816533145308497 1loss/SRqqt1/detection_loss‚1.7561421660440308 1loss/SRqqt4/detection_loss‚0.9786790083679888 2loss/SRqqt2/assignment_loss‚1.9353235432734857 1loss/FBt2/assignment_loss‚0.34614495527299977 1loss/FBt3/assignment_loss‚0.41615557577461004 1loss/SRqqt2/detection_loss‚2.5978448448272853 /loss/FBt1/detection_loss‚2.0559469644839945 /loss/FBt4/detection_loss‚0.8056339812415823 /loss/FRt2/assignment_loss‚3.561898519011105 2loss/SRqqt4/assignment_loss‚1.0024102498880691 .loss/FRt4/detection_loss‚1.440003479968329 0loss/FRt4/assignment_loss‚1.5452438397485702 $loss/total_loss‚40.8023697733901 _runtime‚9320.051164789EøÝ_—*Ù 1loss/SRqqt2/detection_loss‚2.5978448448272853 /loss/FBt1/detection_loss‚2.0559469644839945 /loss/FBt4/detection_loss‚0.8056339812415823 /loss/FRt2/assignment_loss‚3.561898519011105 2loss/SRqqt4/assignment_loss‚1.0024102498880691 .loss/FRt4/detection_loss‚1.440003479968329 0loss/FRt4/assignment_loss‚1.5452438397485702 $loss/total_loss‚40.8023697733901 _runtime‚9320.051164789 1loss/FBt1/assignment_loss‚0.37693514972208786 .loss/FBt2/detection_loss‚2.769229610761007 _step‚962 $ _timestamp‚1.788425820313731e+09 epoch‚9 trainer/global_step‚11699 0loss/FRt3/assignment_loss‚2.5181640934944154 .loss/FRt1/detection_loss‚2.377264790946529 1loss/SRqqt1/assignment_loss‚1.060406762884841 2loss/SRqqt3/assignment_loss‚1.1270433824864172 .loss/FRt2/detection_loss‚2.773805797100067 1loss/SRqqt3/detection_loss‚1.3842759198612637 .loss/FBt3/detection_loss‚2.229116380214691 /loss/FRt1/assignment_loss‚3.469596754936945 1loss/FBt4/assignment_loss‚0.09345068854893213 /loss/FRt3/detection_loss‚2.1816533145308497 1loss/SRqqt1/detection_loss‚1.7561421660440308 1loss/SRqqt4/detection_loss‚0.9786790083679888 2loss/SRqqt2/assignment_loss‚1.9353235432734857 1loss/FBt2/assignment_loss‚0.34614495527299977 1loss/FBt3/assignment_loss‚0.41615557577461004ÂAW¯’˜*‚(:‡ æìäÔý} gpu.0.gpu‚28.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôÿ`œk™*‚(:à æìäԟ disk.drbd2266.out‚0) proc.memory.percent‚16.43803119659424 disk.drbd1716.out‚0 disk.sda3.out‚ 482.0234375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚39788.1416015625 disk.sda2.in‚0 disk.sda3.in‚ 126.77734375 cpu‚25.34119568117936 network.sent‚ 2.8419581e+07 disk.drbd2265.in‚0 disk.rbd1.out‚4.90625% disk.nvme0n1.out‚39788.1416015625 disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1028.5703125 network.recv‚ 3.365768e+06* disk./.usagePercent‚17.563422231301047 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 482.0234375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚39788.1416015625 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875$ memory_percent‚93.72210502624512# proc.memory.rssMB‚ 2693.20703125$ disk./.usageGB‚261.7076835632324 disk.drbd2266.in‚0 disk.sda.out‚ 482.0234375 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.sda.in‚ 126.77734375 proc.cpu.threads‚17¹Ó¿Âš*‚(:… õìäÔÿ~ gpu.0.gpu‚40.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹\Ýt›*‚(:é õìäÔ¶ö€ disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 482.55078125 disk.sda3.out‚ 482.55078125* disk./.usagePercent‚17.565422418166065 disk.nvme0n1.in‚4.6875 network.recv‚ 3.371846e+06 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.drbd2265.in‚0) proc.memory.percent‚16.43803119659424 network.sent‚ 2.8528243e+07 disk.dm-0.in‚4.6875 disk.sda2.out‚0% disk./.usageGB‚261.73748779296875 disk.drbd2080.in‚0# disk.dm-0.out‚39833.63818359375 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1029.06640625 disk.rbd1.in‚0 disk.rbd1.out‚4.90625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875( disk.nvme0c0n1.out‚39833.63818359375 cpu‚25.340757438256674 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.out‚ 482.55078125& disk.nvme0n1.out‚39833.63818359375$ memory_percent‚93.71907711029053# proc.memory.rssMB‚ 2693.20703125 disk.drbd2266.out‚0 disk.sda.in‚ 126.77734375 disk.sda1.out‚0áÆý2œ*- _wandbruntime‚9356  _runtime‚9356ço«‹Ç*Á trainer/global_step‚11749 % _timestamp‚1.7884258567454169e+09 _runtime‚ 9356.48298591 _step‚963 & lr-AdamW/pg1‚4.084516490242617e-07 & lr-AdamW/pg2‚4.084516490242617e-07òÄéÌž*Æ _runtime‚ 9356.48298591 _step‚963 & lr-AdamW/pg1‚4.084516490242617e-07 & lr-AdamW/pg2‚4.084516490242617e-07 trainer/global_step‚11749 % _timestamp‚1.7884258567454169e+09Ãp–_·2Ÿ*- _wandbruntime‚9356  _runtime‚9356l% ê' *äO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221774301 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9838593183232 _step‚964 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚692886994825216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99667468 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745194 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚9356.483902039 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170822600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56138788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268078 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155806714 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756212828501504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320464741376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212643034624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99667470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9802373169966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155806256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794339 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944õŸîÍï'¡*éO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9803027607722 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚756212643034624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚99667470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9802373169966 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155806256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221794339 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683047745279488 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 221774301 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9839249545728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9838593183232 _step‚964 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392596899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚692886994825216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚99667468 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚692886320459776 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674953823745194 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚458 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 _runtime‚9356.483902039 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170822600 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 170822600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56138788 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674953152268078 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683047745279488 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155806714 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756212828501504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10320464741376 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚683047727276544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 185466880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚692886994825216 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680Ää ©2¢*- _wandbruntime‚9357  _runtime‚9357ã Ê2ô&£*îM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155819064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275351955968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321184333312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56143428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10321337458176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412611072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚9357.426815482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683104098318336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9839412611072 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683104098318336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042894ÈfÇù&¤*óM JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚675009544310926 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9803844042894 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155819064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚756275351955968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745953990557696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10321184333312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9803189604930 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392649435 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 170836614 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 221792755 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56143428 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10321337458176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚458 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392629369 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚692944167305216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9839412611072 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170836614 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221812819 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221812819 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _runtime‚9357.426815482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚683104098318336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170836616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9839412611072 _step‚965 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚665205683228672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221792755 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745954014497792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683104098318336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414Åʨ³2¥*- _wandbruntime‚9357  _runtime‚9357 ½z˜W¦*Ñ 1loss/SRqqt1/detection_loss‚1.8882905094122344 1loss/SRqqt4/detection_loss‚0.9616359666872182 .loss/FBt3/detection_loss‚3.051205635070801 # _timestamp‚1.78842585769062e+09 /loss/FRt2/detection_loss‚2.7070648786094456 /loss/FRt1/assignment_loss‚2.978117945947145 2loss/SRqqt2/assignment_loss‚1.7540597514859562 1loss/SRqqt2/detection_loss‚2.7064921527073302 /loss/FBt3/assignment_loss‚0.860794329084456 0loss/FBt4/assignment_loss‚0.1229901648488051 .loss/FRt1/detection_loss‚2.465392135475811 2loss/SRqqt1/assignment_loss‚1.4471000541551886 2loss/SRqqt4/assignment_loss‚0.9379639493142206 0loss/FBt1/assignment_loss‚0.7077772323042154 _step‚966 2loss/SRqqt3/assignment_loss‚1.4427722680257327 .loss/FBt1/detection_loss‚2.637838292121887 .loss/FBt2/detection_loss‚2.262797848744826 epoch‚9 0loss/FRt4/assignment_loss‚1.2869537737634447 0loss/FBt2/assignment_loss‚0.2506229852697262 1loss/SRqqt3/detection_loss‚1.4470218773653258 0loss/FRt3/assignment_loss‚2.8412112374054757 /loss/FBt4/detection_loss‚0.5543219893741863 %loss/total_loss‚43.02823287452989 trainer/global_step‚11749 _runtime‚9357.428103303 0loss/FRt2/assignment_loss‚3.9181106372012033 .loss/FRt3/detection_loss‚2.008262557157299 /loss/FRt4/detection_loss‚1.7894347029979583[éöo\§*Ö .loss/FRt3/detection_loss‚2.008262557157299 /loss/FRt4/detection_loss‚1.7894347029979583 1loss/SRqqt1/detection_loss‚1.8882905094122344 1loss/SRqqt4/detection_loss‚0.9616359666872182 .loss/FBt3/detection_loss‚3.051205635070801 # _timestamp‚1.78842585769062e+09 /loss/FRt2/detection_loss‚2.7070648786094456 /loss/FRt1/assignment_loss‚2.978117945947145 2loss/SRqqt2/assignment_loss‚1.7540597514859562 1loss/SRqqt2/detection_loss‚2.7064921527073302 /loss/FBt3/assignment_loss‚0.860794329084456 0loss/FBt4/assignment_loss‚0.1229901648488051 .loss/FRt1/detection_loss‚2.465392135475811 2loss/SRqqt1/assignment_loss‚1.4471000541551886 2loss/SRqqt4/assignment_loss‚0.9379639493142206 0loss/FBt1/assignment_loss‚0.7077772323042154 _step‚966 2loss/SRqqt3/assignment_loss‚1.4427722680257327 .loss/FBt1/detection_loss‚2.637838292121887 .loss/FBt2/detection_loss‚2.262797848744826 epoch‚9 0loss/FRt4/assignment_loss‚1.2869537737634447 0loss/FBt2/assignment_loss‚0.2506229852697262 1loss/SRqqt3/detection_loss‚1.4470218773653258 0loss/FRt3/assignment_loss‚2.8412112374054757 /loss/FBt4/detection_loss‚0.5543219893741863 %loss/total_loss‚43.02823287452989 trainer/global_step‚11749 _runtime‚9357.428103303 0loss/FRt2/assignment_loss‚3.9181106372012033Æ\û»ÿ¨*‚(:’ „íäÔʼnˆ~ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.969' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.793799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z-³a©*‚(:Ö „íäÔ™Œ¦€ disk.drbd2080.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0$ disk./.usageGB‚261.7567672729492 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 483.1015625 disk.nvme0n1.in‚4.6875! disk.nvme0n1.out‚ 39894.734375 disk.rbd0.out‚0$ memory_percent‚93.71926784515381# proc.memory.rssMB‚ 2693.20703125 disk.sda3.out‚ 483.1015625 disk.rbd1.out‚4.90625 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.43803119659424 network.recv‚ 3.376698e+06 disk.drbd2265.out‚0 disk.dm-0.out‚ 39894.734375 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1029.03515625 network.sent‚ 2.8534529e+07 disk.sda1.in‚0 disk.drbd1716.in‚0# disk.nvme0c0n1.out‚ 39894.734375 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.rbd1.in‚0 disk.sda.out‚ 483.1015625 disk.drbd1716.out‚0 cpu‚25.340238442737416* disk./.usagePercent‚17.566716280244115é°Ô`’ª*‚(:‡ “íäÔÀ¥¢~ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚176.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0às.Ûf«*‚(:Û “íäԺϞ disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.dm-1.out‚ 483.49609375 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚4.6875 disk.dm-0.out‚39939.66015625 disk.drbd2266.out‚0 disk.rbd1.out‚4.90625) proc.memory.percent‚16.43803119659424 disk.sda2.in‚0 disk.sda3.out‚ 483.49609375 disk.sda.out‚ 483.49609375# proc.memory.rssMB‚ 2693.20703125 disk.dm-1.in‚ 126.77734375$ disk./.usageGB‚261.7763977050781 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.sda.in‚ 126.77734375 network.sent‚ 2.864235e+07# disk.nvme0n1.out‚39939.66015625 disk.nvme0c0n1.in‚4.6875% disk.nvme0c0n1.out‚39939.66015625 disk.drbd2265.out‚0 disk.drbd2266.in‚0& proc.memory.availableMB‚ 1029.03125 cpu‚25.339796704552654 network.recv‚ 3.381875e+06* disk./.usagePercent‚17.568033695015316$ memory_percent‚93.71929168701172 proc.cpu.threads‚17+Th’¬*‚(:‡ ¢íäÔÔû¦~ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´fÇr­*‚(:ç ¢íäÔÇɪ€* disk./.usagePercent‚17.567471246462592% disk./.usageGB‚261.76801681518555 disk.drbd2265.out‚0# disk.dm-0.out‚39994.07763671875 disk.dm-1.in‚ 126.77734375 disk.drbd1716.out‚0 disk.sda3.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.out‚0 disk.drbd2266.in‚0# memory_percent‚93.7213659286499 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚4.6875# proc.memory.rssMB‚ 2693.20703125& disk.nvme0n1.out‚39994.07763671875 network.sent‚ 2.8648441e+07 network.recv‚ 3.386439e+06 disk.dm-1.out‚ 484.05859375 disk.sda2.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚39994.07763671875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda.out‚ 484.05859375 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4.6875 disk.drbd2266.out‚0 disk.rbd1.out‚4.90625 disk.sda.in‚ 126.77734375 disk.sda3.out‚ 484.05859375) proc.memory.availableMB‚ 1028.69140625 cpu‚25.33931354013928 proc.cpu.threads‚17 disk.rbd0.in‚0) proc.memory.percent‚16.43803119659424ô½èŠ2®*- _wandbruntime‚9393  _runtime‚9393j>ôEʯ*Ä ' lr-AdamW/pg1‚2.9751798364348915e-07 ' lr-AdamW/pg2‚2.9751798364348915e-07 trainer/global_step‚11799 % _timestamp‚1.7884258941901867e+09 _runtime‚9393.927701185 _step‚967è1/—Ï°*É ' lr-AdamW/pg1‚2.9751798364348915e-07 ' lr-AdamW/pg2‚2.9751798364348915e-07 trainer/global_step‚11799 % _timestamp‚1.7884258941901867e+09 _runtime‚9393.927701185 _step‚967ÇlÍÚ©2±*- _wandbruntime‚9393  _runtime‚9393;ÛAìDz*åO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171523398 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156443879 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171523398 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9879578989056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9879578989056 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 191758336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748974876703232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685865397221888 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚695745632573440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10363511769600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 171523400 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748974900643328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56368463 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156444333 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100075870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56368011 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612602 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667896002665472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222720391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10363343951360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884258941908793e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667895985626112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685865397221888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759338220654592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677739865612602 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677739194135486 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9843862947130 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚667896002665472 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222720391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10363343951360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884258941908793e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚695744958208000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚667895985626112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685865397221888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚452 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚759338220654592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222700353 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 167818240 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚695744958208000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685865379218944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 394223751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56368463 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156444333 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100075870 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56368011 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222718807 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156457017 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130674688 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9880398416896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9880398416896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805053440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749036527440896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9881054792704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156456561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677794914701010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 394276287 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10364195710976 _runtime‚9394.625939831 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788425894887714e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222738871 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178334 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚759400914896896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0AaV¢¶*ôM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 191745024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9880398416896 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚685921732257792 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚695802805053440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100084038 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749036527440896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9881054792704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156456561 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚677794914701010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 394276287 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 394256221 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10364195710976 _runtime‚9394.625939831 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685921750260736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 $ _timestamp‚1.788425894887714e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167804928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚685921750260736 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222738871 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677795586178334 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train¢::‘X _batch_end/inactive_split_bytes.all.allocated‚759400914896896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9844679382302 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171537416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749036551380992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 394256221 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222718807 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9881054792704 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156457017 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 394276287 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 171537414 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11799 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚695802130674688 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171537414 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100084036 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚667950906796032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685921732257792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222738871 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚695802805053440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9880398416896 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680É´ÈC2·*- _wandbruntime‚9394  _runtime‚9394¡±Z¸*Ô _step‚970 /loss/FRt3/assignment_loss‚2.984901284917872 /loss/FRt4/assignment_loss‚1.265501100707937 1loss/FBt4/assignment_loss‚0.11426806443514012 /loss/FRt3/detection_loss‚2.3604719911484007 %loss/total_loss‚40.32208938056092 % _timestamp‚1.7884258948893876e+09 /loss/FRt1/detection_loss‚2.3839468186100325 0loss/SRqqt1/detection_loss‚1.895872479373658 trainer/global_step‚11799 /loss/FRt2/assignment_loss‚3.173781238934573 1loss/SRqqt2/assignment_loss‚1.555077002598689 1loss/FBt3/assignment_loss‚0.16510565413368994 /loss/FBt2/detection_loss‚2.3442447602748873 /loss/FBt4/detection_loss‚0.7453664891313898 .loss/FBt3/detection_loss‚2.649087733692593 /loss/FRt4/detection_loss‚1.4412517420671604 /loss/FRt1/assignment_loss‚2.988447339998351 2loss/SRqqt4/assignment_loss‚0.8105051679007899 0loss/SRqqt2/detection_loss‚2.733219113487464 epoch‚9 2loss/SRqqt1/assignment_loss‚1.1347355307252318 1loss/SRqqt3/assignment_loss‚1.376736533092825 1loss/FBt2/assignment_loss‚0.05852751474303659 1loss/SRqqt4/detection_loss‚0.9311733827373739 /loss/FBt1/detection_loss‚2.4845990190903344 _runtime‚9394.626891958 0loss/FBt1/assignment_loss‚0.2840862402420801 /loss/FRt2/detection_loss‚2.7202810329549454 1loss/SRqqt3/detection_loss‚1.7209021455624647AKñ8_¹*Ù /loss/FRt1/assignment_loss‚2.988447339998351 2loss/SRqqt4/assignment_loss‚0.8105051679007899 0loss/SRqqt2/detection_loss‚2.733219113487464 epoch‚9 2loss/SRqqt1/assignment_loss‚1.1347355307252318 1loss/SRqqt3/assignment_loss‚1.376736533092825 1loss/FBt2/assignment_loss‚0.05852751474303659 1loss/SRqqt4/detection_loss‚0.9311733827373739 /loss/FBt1/detection_loss‚2.4845990190903344 _runtime‚9394.626891958 0loss/FBt1/assignment_loss‚0.2840862402420801 /loss/FRt2/detection_loss‚2.7202810329549454 1loss/SRqqt3/detection_loss‚1.7209021455624647 _step‚970 /loss/FRt3/assignment_loss‚2.984901284917872 /loss/FRt4/assignment_loss‚1.265501100707937 1loss/FBt4/assignment_loss‚0.11426806443514012 /loss/FRt3/detection_loss‚2.3604719911484007 %loss/total_loss‚40.32208938056092 % _timestamp‚1.7884258948893876e+09 /loss/FRt1/detection_loss‚2.3839468186100325 0loss/SRqqt1/detection_loss‚1.895872479373658 trainer/global_step‚11799 /loss/FRt2/assignment_loss‚3.173781238934573 1loss/SRqqt2/assignment_loss‚1.555077002598689 1loss/FBt3/assignment_loss‚0.16510565413368994 /loss/FBt2/detection_loss‚2.3442447602748873 /loss/FBt4/detection_loss‚0.7453664891313898 .loss/FBt3/detection_loss‚2.649087733692593 /loss/FRt4/detection_loss‚1.4412517420671604ʤ(ƒp’º*‚(:‡ ±íäÔõ®à gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.3178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óf÷l»*‚(:á ±íäÔ¡Þ€( disk.nvme0c0n1.out‚40039.30810546875* disk./.usagePercent‚17.568891064247488 disk.drbd2265.in‚0 disk.drbd2266.out‚0& disk.nvme0n1.out‚40039.30810546875) proc.memory.percent‚16.43803119659424 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚4.90625 disk.sda.in‚ 126.77734375 disk.nvme0c0n1.in‚4.6875 network.recv‚ 3.390507e+06$ disk./.usageGB‚261.7891731262207 disk.sda2.out‚0 disk.dm-1.in‚ 126.77734375 disk.sda.out‚ 484.546875$ memory_percent‚93.72174739837646 disk.sda2.in‚0 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 484.546875# disk.dm-0.out‚40039.30810546875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1028.62890625 proc.cpu.threads‚17 disk.dm-0.in‚4.6875# proc.memory.rssMB‚ 2693.20703125 cpu‚25.338851112313222 disk.dm-1.out‚ 484.546875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 2.865442e+07 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0feÌ…’¼*‚(:‡ ÀíäÔû“¢~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚184.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.9308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôç¥×e½*‚(:Ú ÀíäÔ‚½³ disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0( proc.memory.availableMB‚ 1028.6328125 network.recv‚ 3.397598e+06 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.sda2.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.43803119659424& disk.nvme0n1.out‚40098.66943359375 disk.rbd0.in‚0 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875 disk.sda3.out‚485.1875 disk.drbd2080.in‚0 cpu‚25.338415265067468 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚40098.66943359375 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375# disk.dm-0.out‚40098.66943359375 disk.rbd1.out‚4.90625 disk.sda.out‚485.1875 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚4.6875 disk.dm-1.out‚485.1875 disk.drbd2266.out‚0$ memory_percent‚93.72172355651855# proc.memory.rssMB‚ 2693.20703125 network.sent‚ 2.8763451e+07( disk./.usagePercent‚17.5683962016838% disk./.usageGB‚261.78179931640625 disk.sda1.in‚0 disk.rbd0.out‚0m¾Øm2¾*- _wandbruntime‚9431  _runtime‚9431ÕP×Ç¿*Á & lr-AdamW/pg1‚2.040993339706666e-07 & lr-AdamW/pg2‚2.040993339706666e-07 trainer/global_step‚11849 $ _timestamp‚1.788425931333306e+09 _runtime‚9431.070971538 _step‚971£´H™ÌÀ*Æ _step‚971 & lr-AdamW/pg1‚2.040993339706666e-07 & lr-AdamW/pg2‚2.040993339706666e-07 trainer/global_step‚11849 $ _timestamp‚1.788425931333306e+09 _runtime‚9431.070971538Ëy ˆ«2Á*- _wandbruntime‚9431  _runtime‚9431ÛoBêì'Â*æO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646443 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172224200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157082716 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚698604270321664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9920564794880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762463885565952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11849 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9884043848782 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚9431.071898447 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406448039424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223626405 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564794880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56598891 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270321664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670641192154112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157083161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762463706390528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752057413586432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56598448 _step‚972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698603595956224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0•xa‚ñ'Ã*ëO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚443 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚698604270321664 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395870643 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9920564794880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚762463885565952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 trainer/global_step‚11849 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 172224200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9884698286538 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9884043848782 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _runtime‚9431.071898447 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10406448039424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223626405 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9920564794880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56598891 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698604270321664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚670641192154112 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157083161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚445 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172224198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚762463706390528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 172224198 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚752057413586432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56598448 _step‚972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚680525907480010 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚698603595956224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698603595956224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9921221157376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223646443 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚688683031161344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚688683049164288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9921221157376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚688683049164288 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223646443 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172224200 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 157082716Ìå’#t2Ä*- _wandbruntime‚9431  _runtime‚9431mƒËëR Å*ïM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688739384200192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688739402203136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680580956568418 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9922040598528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100492438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/activ”š­O£e.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157095545 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157095995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223664923 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407144940544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442801664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚11849 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407310648320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56603557 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430ý?`ú&Æ*ôM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9921384222720 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172238214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223644859 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 9326034944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚698661442801664 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 trainer/global_step‚11849 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚752119064324096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10407310648320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9884860283746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚752119088264192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223664923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚688739402203136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56603557 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223644859 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚688739384200192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚688739402203136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9885514721710 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚698660768422912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚680580956568418 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9922040598528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395903139 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100492438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670696096284672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157095545 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395903139 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157095995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223664923 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10407144940544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395883073Í‘åº>2Ç*- _wandbruntime‚9431  _runtime‚9431Q¤LYXÈ*Ò 2loss/SRqqt4/assignment_loss‚0.8494736182713701 -loss/FRt4/detection_loss‚1.82846836183431 1loss/SRqqt3/detection_loss‚1.5699359361897223 epoch‚9 1loss/SRqqt4/detection_loss‚1.0809227695027668 $ _timestamp‚1.788425932031522e+09 /loss/FRt1/assignment_loss‚2.811826745669047 _step‚974 2loss/SRqqt2/assignment_loss‚1.4922710294812553 0loss/FBt3/assignment_loss‚0.2587599270045568 2loss/SRqqt3/assignment_loss‚1.3404334751539864 0loss/FBt2/assignment_loss‚0.4943971209570877 /loss/FBt4/detection_loss‚0.5123952742184649 _runtime‚9431.768972328 0loss/FRt2/assignment_loss‚3.0711655712127683 0loss/FRt3/assignment_loss‚2.3090353186537578 2loss/SRqqt1/assignment_loss‚1.4729757687076925 /loss/FRt2/detection_loss‚3.0483895111083985 0loss/FRt4/assignment_loss‚1.0269419427187938 1loss/FBt4/assignment_loss‚0.11113555604320743 .loss/FRt3/detection_loss‚2.455444541646213 .loss/FBt1/detection_loss‚2.639317672986251 .loss/FBt2/detection_loss‚2.690832083875483 /loss/FBt3/detection_loss‚3.1902852058410645 %loss/total_loss‚41.80387615776236 1loss/FBt1/assignment_loss‚0.11162271921952702 -loss/FRt1/detection_loss‚2.91042873164018 1loss/SRqqt1/detection_loss‚2.0928903198904463 0loss/SRqqt2/detection_loss‚2.434526955936013 trainer/global_step‚11849ú:Ü]É*× -loss/FRt4/detection_loss‚1.82846836183431 1loss/SRqqt3/detection_loss‚1.5699359361897223 epoch‚9 1loss/SRqqt4/detection_loss‚1.0809227695027668 $ _timestamp‚1.788425932031522e+09 /loss/FRt1/assignment_loss‚2.811826745669047 _step‚974 2loss/SRqqt2/assignment_loss‚1.4922710294812553 0loss/FBt3/assignment_loss‚0.2587599270045568 2loss/SRqqt3/assignment_loss‚1.3404334751539864 0loss/FBt2/assignment_loss‚0.4943971209570877 /loss/FBt4/detection_loss‚0.5123952742184649 _runtime‚9431.768972328 0loss/FRt2/assignment_loss‚3.0711655712127683 0loss/FRt3/assignment_loss‚2.3090353186537578 2loss/SRqqt1/assignment_loss‚1.4729757687076925 /loss/FRt2/detection_loss‚3.0483895111083985 0loss/FRt4/assignment_loss‚1.0269419427187938 1loss/FBt4/assignment_loss‚0.11113555604320743 .loss/FRt3/detection_loss‚2.455444541646213 .loss/FBt1/detection_loss‚2.639317672986251 .loss/FBt2/detection_loss‚2.690832083875483 /loss/FBt3/detection_loss‚3.1902852058410645 %loss/total_loss‚41.80387615776236 1loss/FBt1/assignment_loss‚0.11162271921952702 -loss/FRt1/detection_loss‚2.91042873164018 1loss/SRqqt1/detection_loss‚2.0928903198904463 0loss/SRqqt2/detection_loss‚2.434526955936013 trainer/global_step‚11849 2loss/SRqqt4/assignment_loss‚0.8494736182713701δ§¡’Ê*‚(:‡ ÏíäÔºï¦~ gpu.0.gpu‚29.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.158' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™†¢kË*‚(:à ÏíäÔØÍ $ disk./.usageGB‚261.8017921447754 disk.sda3.in‚ 126.77734375# disk.dm-0.out‚40144.49755859375 disk.drbd1716.out‚0& disk.nvme0n1.out‚40144.49755859375& proc.memory.availableMB‚ 1028.84375 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 485.5859375 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2693.20703125 cpu‚25.337932389724163 network.sent‚ 2.8768613e+07 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875# memory_percent‚93.7204360961914 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.40107e+06 disk.rbd0.in‚0( disk.nvme0c0n1.out‚40144.49755859375 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 485.5859375* disk./.usagePercent‚17.569737937170753 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375 disk.sda3.out‚ 485.5859375 disk.drbd2266.in‚0 disk.rbd1.out‚4.90625 disk.sda1.out‚0) proc.memory.percent‚16.438031196594240Ñ?2’Ì*‚(:‡ ÞíäÔ“þ©~ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚175.407' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.0814 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̵XDlÍ*‚(:á ÞíäÔ­Þ±€ disk.drbd2080.out‚0 network.sent‚ 2.887748e+07$ disk./.usageGB‚261.8041191101074 disk.rbd0.in‚0' disk.nvme0c0n1.out‚40198.3056640625 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 486.24609375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚4.6875) proc.memory.availableMB‚ 1028.34765625# proc.memory.rssMB‚ 2693.20703125 cpu‚25.337507763015296) disk./.usagePercent‚17.56989410176664 disk.drbd2266.in‚0 disk.sda.in‚ 126.77734375) proc.memory.percent‚16.43803119659424 proc.cpu.threads‚17" disk.dm-0.out‚40198.3056640625 disk.dm-1.out‚ 486.24609375 disk.drbd2266.out‚0 disk.rbd1.out‚4.90625 disk.drbd2080.in‚0" memory_percent‚93.723464012146 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.sda.out‚ 486.24609375% disk.nvme0n1.out‚40198.3056640625 network.recv‚ 3.407458e+06 disk.nvme0c0n1.in‚4.6875 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 disk.sda1.in‚0 disk.sda1.out‚0@ž¦’Î*‚(:‡ ííäÔÙ¡²~ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.627' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.9254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04wIClÏ*‚(:á ííäÔÀÉÿ€ disk.nvme0n1.in‚4.6875) proc.memory.availableMB‚ 1028.59765625 cpu‚25.337049973784747' disk.nvme0c0n1.out‚40241.7744140625 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 486.7890625" disk.dm-0.out‚40241.7744140625 disk.sda.out‚ 486.7890625 disk.sda2.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚40241.7744140625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0c0n1.in‚4.6875 disk.rbd1.in‚0 disk.dm-1.out‚ 486.7890625 disk.drbd2266.out‚0 network.sent‚ 2.888329e+07 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚4.90625 disk.sda.in‚ 126.77734375$ memory_percent‚93.72193813323975) proc.memory.percent‚16.43803119659424 network.recv‚ 3.411634e+06 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚17.570974197553603 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2693.20703125$ disk./.usageGB‚261.8202133178711 disk.rbd0.out‚0 disk.dm-0.in‚4.6875 disk.dm-1.in‚ 126.77734375'Þ[Í2Ð*- _wandbruntime‚9468  _runtime‚9468jh'¹ÈÑ* _step‚975 & lr-AdamW/pg1‚1.282151683123586e-07 & lr-AdamW/pg2‚1.282151683123586e-07 trainer/global_step‚11899 % _timestamp‚1.7884259684652216e+09 _runtime‚9468.202775891ýTJÍÒ*Ç trainer/global_step‚11899 % _timestamp‚1.7884259684652216e+09 _runtime‚9468.202775891 _step‚975 & lr-AdamW/pg1‚1.282151683123586e-07 & lr-AdamW/pg2‚1.282151683123586e-07ÏS©Â‡2Ó*- _wandbruntime‚9468  _runtime‚9468ƈ-|Ô*æO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10449623133696 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9961550600704 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224552457 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚451 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 9955586048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9962206963200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157722185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100892670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100892668 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172924998 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚9468.203703642 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56829515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589597543424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9925533625946 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚691500683103744 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9962206963200 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224572495 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157722185 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚701462233704448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172925000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100892670 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100892668 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 172924998 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 397497495 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9924879188190 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚430 _runtime‚9468.203703642 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397477455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚453 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56829515 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 165721088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 868220928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚701462908069888 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 189661184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚765589597543424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9925533625946 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚701462233704448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 8838745088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157734415 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765651927188480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚977 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11899 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224570911 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026404352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570911 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157734864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435826 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691557036142592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691557054145536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590975 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061118 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605-£”7ò&Ø*õM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9963026404352 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 157734864 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚683366998435826 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673441319852032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9962370028544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691557036142592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397529991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 674378752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 183356416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚755201601207296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691557054145536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 172939014 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397509925 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224590975 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9926350061118 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9962370028544 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100900836 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚755201625147392 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚701520080549888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 397529991 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691557054145536 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157734415 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚765651927188480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚977 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚908 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚844 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224590975 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 trainer/global_step‚11899 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224570911 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100900838 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚447 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9963026404352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 172939014 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224570911 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated ·¤Ì ‚0Ñ*œäã2Ù*- _wandbruntime‚9468  _runtime‚9468 -ÿWÚ*Ñ 0loss/FBt1/assignment_loss‚0.1619788079776547 0loss/FBt2/assignment_loss‚0.7572542379299231 .loss/FRt1/detection_loss‚2.604622507417524 $loss/total_loss‚41.6116510105069 2loss/SRqqt2/assignment_loss‚1.5918235144600636 1loss/SRqqt4/detection_loss‚0.8836282396034615 .loss/FBt1/detection_loss‚2.779071255163713 /loss/FBt3/detection_loss‚2.4700716000336866 epoch‚9 % _timestamp‚1.7884259691631672e+09 1loss/SRqqt1/detection_loss‚1.7341350483206601 0loss/SRqqt2/detection_loss‚2.326121804190845 /loss/FRt2/detection_loss‚2.9688685964792967 /loss/FRt3/assignment_loss‚2.581119508159404 2loss/SRqqt1/assignment_loss‚1.3970551934236517 1loss/FBt3/assignment_loss‚0.09315098070412485 1loss/FBt4/assignment_loss‚0.16999783733435492 -loss/FRt3/detection_loss‚2.09923887617734 1loss/SRqqt3/detection_loss‚1.8741675704993583 .loss/FBt2/detection_loss‚2.926903943220774 /loss/FBt4/detection_loss‚0.6479313025627745 .loss/FRt1/assignment_loss‚3.18867425338642 /loss/FRt2/assignment_loss‚3.015249812975526 0loss/FRt4/assignment_loss‚1.4244969759071082 _runtime‚9468.900615226 2loss/SRqqt3/assignment_loss‚1.3330077362005357 2loss/SRqqt4/assignment_loss‚1.0350479041383527 /loss/FRt4/detection_loss‚1.5480335042403455 trainer/global_step‚11899 _step‚978m<Í\Û*Ö /loss/FBt4/detection_loss‚0.6479313025627745 .loss/FRt1/assignment_loss‚3.18867425338642 /loss/FRt2/assignment_loss‚3.015249812975526 0loss/FRt4/assignment_loss‚1.4244969759071082 _runtime‚9468.900615226 2loss/SRqqt3/assignment_loss‚1.3330077362005357 2loss/SRqqt4/assignment_loss‚1.0350479041383527 /loss/FRt4/detection_loss‚1.5480335042403455 trainer/global_step‚11899 _step‚978 0loss/FBt1/assignment_loss‚0.1619788079776547 0loss/FBt2/assignment_loss‚0.7572542379299231 .loss/FRt1/detection_loss‚2.604622507417524 $loss/total_loss‚41.6116510105069 2loss/SRqqt2/assignment_loss‚1.5918235144600636 1loss/SRqqt4/detection_loss‚0.8836282396034615 .loss/FBt1/detection_loss‚2.779071255163713 /loss/FBt3/detection_loss‚2.4700716000336866 epoch‚9 % _timestamp‚1.7884259691631672e+09 1loss/SRqqt1/detection_loss‚1.7341350483206601 0loss/SRqqt2/detection_loss‚2.326121804190845 /loss/FRt2/detection_loss‚2.9688685964792967 /loss/FRt3/assignment_loss‚2.581119508159404 2loss/SRqqt1/assignment_loss‚1.3970551934236517 1loss/FBt3/assignment_loss‚0.09315098070412485 1loss/FBt4/assignment_loss‚0.16999783733435492 -loss/FRt3/detection_loss‚2.09923887617734 1loss/SRqqt3/detection_loss‚1.8741675704993583 .loss/FBt2/detection_loss‚2.926903943220774Ò¢ýOÜ*‚(:… üíäÔ믾~ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚36.0 gpu.0.powerWatts‚138.49' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qW¢CoÝ*‚(:ä üíäÔ’¬í€ disk.dm-1.in‚ 126.77734375 cpu‚25.336571558562596 network.sent‚ 2.8889679e+07 disk.rbd0.out‚0" disk.dm-0.out‚40299.7646484375 disk.dm-1.out‚ 487.46484375 disk.drbd2266.in‚0 disk.rbd1.out‚4.90625 disk.drbd1716.in‚0 disk.sda3.in‚ 126.77734375 disk.dm-0.in‚4.6875 disk.nvme0n1.in‚4.6875 disk.rbd0.in‚0 disk.sda3.out‚ 487.46484375) proc.memory.percent‚16.43803119659424 network.recv‚ 3.416519e+06* disk./.usagePercent‚17.572524835188492 disk.drbd2265.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.20703125 proc.cpu.threads‚17 disk.sda.out‚ 487.46484375 disk.sda1.in‚0# disk./.usageGB‚261.843318939209 disk.nvme0c0n1.in‚4.6875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0% disk.nvme0n1.out‚40299.7646484375) proc.memory.availableMB‚ 1028.04296875' disk.nvme0c0n1.out‚40299.7646484375 disk.drbd2266.out‚0 disk.sda1.out‚0$ memory_percent‚93.72532367706299â¸ï’Þ*‚(:‡ ‹îäÔ‚¨€~ gpu.0.gpu‚44.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S­¯Ejß*‚(:ß ‹îäÔù‡ disk.sda3.in‚ 126.77734375$ memory_percent‚93.72231960296631% disk./.usageGB‚261.85933685302734 disk.dm-1.in‚ 126.77734375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 487.78125 disk.rbd1.out‚4.90625 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2693.20703125 disk.rbd1.in‚0 disk.sda.out‚ 487.78125 disk.drbd1716.in‚0 disk.nvme0n1.in‚4.6875 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1028.53515625 cpu‚25.336119009736617 network.sent‚ 2.8997611e+07) disk./.usagePercent‚17.57359981082477 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚4.6875# disk.dm-0.out‚40356.95361328125) proc.memory.percent‚16.43803119659424 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚40356.95361328125 disk.sda2.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.422184e+06 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 487.78125 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375& disk.nvme0n1.out‚40356.95361328125 disk.rbd0.out‚0Fj‡Á2à*- _wandbruntime‚9505  _runtime‚9505±t"Çá*Á trainer/global_step‚11949 % _timestamp‚1.7884260057560432e+09 _runtime‚ 9505.49353551 _step‚979 & lr-AdamW/pg1‚6.988130081497319e-08 & lr-AdamW/pg2‚6.988130081497319e-08\¾9Ìâ*Æ & lr-AdamW/pg2‚6.988130081497319e-08 trainer/global_step‚11949 % _timestamp‚1.7884260057560432e+09 _runtime‚ 9505.49353551 _step‚979 & lr-AdamW/pg1‚6.988130081497319e-08Ó2‹2ã*- _wandbruntime‚9505  _runtime‚9505N¬'^ð'ä*êO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9965714527598 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101301070 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10492607916544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚694318335046144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10492777831936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002536406528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871452672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225478509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚704320871452672 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225478509 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169915392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10492607916544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚694318335046144 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10492777831936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚465 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002536406528 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚686097319737710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚704320871452672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768777790106624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493460271104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173639816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173639816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225496963 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101309238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158374091 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158373647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704378718298112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57064411 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173639814 L>DeviceStatsMonitor.on_train_batch_end/a.ß@û_ctive_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9506.197595329 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704378718298112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚758284138090496|2Þý&è*÷M LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 _step‚981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚704378718298112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101309236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57064411 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9967185400526 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 173639814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 _runtime‚9506.197595329 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚704378718298112 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 399136777 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 399156843 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚758284138090496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚704378043919360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚444 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768777790106624 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚442 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10493460271104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 399156843 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 173639816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11949 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173639816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225496963 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101309238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158374091 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158373647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 10194255872Õ;4˜Ù2é*- _wandbruntime‚9506  _runtime‚9506I7U*Zê*Ô 0loss/FRt2/assignment_loss‚3.2644473129305345 0loss/FBt1/assignment_loss‚0.4578356414567679 /loss/FRt1/detection_loss‚2.5416477965681175 1loss/SRqqt2/assignment_loss‚1.267383557278663 2loss/SRqqt3/assignment_loss‚1.5755894935822912 1loss/FBt2/assignment_loss‚0.13827820958717243 /loss/FRt2/detection_loss‚2.8532059274870774 0loss/SRqqt3/detection_loss‚1.435635445772537 .loss/FRt3/detection_loss‚2.584231828649839 %loss/total_loss‚40.29179476795221 2loss/SRqqt4/assignment_loss‚0.8415584063820415 /loss/FBt4/detection_loss‚0.5521710628678126 _step‚982 1loss/SRqqt1/detection_loss‚2.3550244217117626 1loss/SRqqt4/detection_loss‚0.7712500754554393 /loss/FBt3/detection_loss‚2.5794073939323425 0loss/FRt1/assignment_loss‚3.1790190056750647 0loss/FRt4/assignment_loss‚1.0540814030738104 1loss/SRqqt1/assignment_loss‚1.252715894797196 1loss/FBt4/assignment_loss‚0.18432093671989053 1loss/SRqqt2/detection_loss‚2.1868812199681997 /loss/FBt2/detection_loss‚2.6361806392669678 trainer/global_step‚11949 $ _timestamp‚1.788426006461195e+09 /loss/FBt3/assignment_loss‚1.079429779124041 /loss/FBt1/detection_loss‚1.7796885669231415 epoch‚9 _runtime‚9506.198631385 /loss/FRt3/assignment_loss‚2.286011662748125 .loss/FRt4/detection_loss‚1.435799085993379¯Ð¦ƒ_ë*Ù 2loss/SRqqt4/assignment_loss‚0.8415584063820415 /loss/FBt4/detection_loss‚0.5521710628678126 _step‚982 1loss/SRqqt1/detection_loss‚2.3550244217117626 1loss/SRqqt4/detection_loss‚0.7712500754554393 /loss/FBt3/detection_loss‚2.5794073939323425 0loss/FRt1/assignment_loss‚3.1790190056750647 0loss/FRt4/assignment_loss‚1.0540814030738104 1loss/SRqqt1/assignment_loss‚1.252715894797196 1loss/FBt4/assignment_loss‚0.18432093671989053 1loss/SRqqt2/detection_loss‚2.1868812199681997 /loss/FBt2/detection_loss‚2.6361806392669678 trainer/global_step‚11949 $ _timestamp‚1.788426006461195e+09 /loss/FBt3/assignment_loss‚1.079429779124041 /loss/FBt1/detection_loss‚1.7796885669231415 epoch‚9 _runtime‚9506.198631385 /loss/FRt3/assignment_loss‚2.286011662748125 .loss/FRt4/detection_loss‚1.435799085993379 0loss/FRt2/assignment_loss‚3.2644473129305345 0loss/FBt1/assignment_loss‚0.4578356414567679 /loss/FRt1/detection_loss‚2.5416477965681175 1loss/SRqqt2/assignment_loss‚1.267383557278663 2loss/SRqqt3/assignment_loss‚1.5755894935822912 1loss/FBt2/assignment_loss‚0.13827820958717243 /loss/FRt2/detection_loss‚2.8532059274870774 0loss/SRqqt3/detection_loss‚1.435635445772537 .loss/FRt3/detection_loss‚2.584231828649839 %loss/total_loss‚40.29179476795221Ö{â0t’ì*‚(:‡ šîäÔ®ˆ¤~ gpu.0.gpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚178.124' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.6248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ãr]í*‚(:Ò šîäÔ‡‹º€) proc.memory.percent‚16.43803119659424 disk.drbd2266.in‚0 network.sent‚ 2.9003623e+07) disk./.usagePercent‚17.57450121335283 disk.drbd2080.in‚0 disk.dm-1.out‚ 488.484375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 126.77734375 disk.rbd1.out‚4.90625 disk.sda.out‚ 488.484375 disk.sda1.out‚0% disk.nvme0c0n1.out‚40410.21484375 disk.sda3.in‚ 126.77734375 network.recv‚ 3.426515e+06 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0# proc.memory.availableMB‚1028.25 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-0.in‚4.6875 cpu‚25.335672220442653$ disk./.usageGB‚261.8727684020996 disk.drbd2080.out‚0 disk.drbd2266.out‚0# disk.nvme0n1.out‚40410.21484375 disk.dm-0.out‚40410.21484375 disk.rbd1.in‚0 disk.sda.in‚ 126.77734375 disk.drbd1716.in‚0$ memory_percent‚93.72406005859375 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 488.484375# proc.memory.rssMB‚ 2693.20703125Å#XÕî*‚(:’ ©îäÔ¹þ} gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.636599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K=?Zï*‚(:Ï ©îäÔ×Ø‹€ network.sent‚ 2.9112102e+07 disk.dm-1.in‚ 126.77734375 disk.dm-1.out‚ 488.84375 disk.drbd2266.in‚0 disk.rbd1.out‚4.90625 disk.sda.in‚ 126.77734375 disk.sda.out‚ 488.84375 disk.drbd1716.in‚0 disk.dm-0.out‚40447.67578125 disk.drbd1716.out‚0% disk.nvme0c0n1.out‚40447.67578125# memory_percent‚93.7225341796875# proc.memory.rssMB‚ 2693.20703125) proc.memory.percent‚16.43803119659424* disk./.usagePercent‚17.575072110154185 disk.drbd2266.out‚0# disk.nvme0n1.out‚40447.67578125 disk.dm-0.in‚4.6875 disk.rbd1.in‚0 disk.nvme0n1.in‚4.6875 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 proc.cpu.threads‚17% disk./.usageGB‚261.88127517700195 disk.drbd2080.in‚0 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 488.84375 disk.drbd2265.in‚0" proc.memory.availableMB‚1028.5 network.recv‚ 3.432283e+06 disk.sda2.out‚0 cpu‚25.335199437549097º¶%#ð*‚(:… ¸îäÔϦ—~ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚181.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;kÛjñ*‚(:ß ¸îäÔø­ç€ disk.sda.in‚ 126.77734375 disk.sda1.in‚0) proc.memory.availableMB‚ 1028.49609375) proc.memory.percent‚16.43803119659424 disk.drbd1716.out‚0 disk.nvme0c0n1.in‚4.6875 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚40501.89013671875 proc.cpu.threads‚17& disk.nvme0n1.out‚40501.89013671875 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.drbd2265.in‚0 disk.dm-0.in‚4.6875$ memory_percent‚93.72255802154541 disk.rbd1.out‚4.90625( disk.nvme0c0n1.out‚40501.89013671875 disk.sda2.in‚0) disk./.usagePercent‚17.57632629106443 disk.nvme0n1.in‚4.6875 disk.drbd2080.in‚0 disk.sda.out‚ 489.59375 cpu‚25.334748292574496 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2693.20703125% disk./.usageGB‚261.89996337890625 disk.drbd1716.in‚0 disk.sda1.out‚0 network.recv‚ 3.437278e+06 disk.rbd0.in‚0 disk.dm-1.out‚ 489.59375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 network.sent‚ 2.9118269e+07 disk.rbd0.out‚0 disk.sda3.out‚ 489.59375 disk.rbd1.in‚0ÓL•2ò*- _wandbruntime‚9542  _runtime‚9542ÏQ¥ÛÊó*Ä ' lr-AdamW/pg1‚2.9109888169040593e-08 ' lr-AdamW/pg2‚2.9109888169040593e-08 trainer/global_step‚11999 % _timestamp‚1.7884260424387882e+09 _runtime‚9542.176503262 _step‚983¾ì[ÓÏô*É ' lr-AdamW/pg2‚2.9109888169040593e-08 trainer/global_step‚11999 % _timestamp‚1.7884260424387882e+09 _runtime‚9542.176503262 _step‚983 ' lr-AdamW/pg1‚2.9109888169040593e-08×(Ý&2õ*- _wandbruntime‚9542  _runtime‚9542°‹‹ò'ö*ìO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚761305048176128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚761305024236032 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522212352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101709470 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174326598 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549867006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697135986988544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178574848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535714587136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404561 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚697136004991488 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚707179509200896 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10043522212352 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 674365440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10043522212352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚688883361605118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101709470 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 174326598 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10006549867006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 671477116 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚697135986988544 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10044178574848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10535714587136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226404561 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 861853521 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400751199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚707180183566336 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚697136004991488 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚985 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678931715868672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159013168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 trainer/global_step‚11999 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226423015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57295532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884260431304832e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159013637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10536593839104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚9542.868403342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707237356046336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57295999 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771903268812800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707237356046336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101717638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761366698913792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424XÁs 'ú*ûM SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101717636 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 861853521 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 174340616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 177064960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 9148669952 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400763629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚771903268812800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚707237356046336 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 174340614 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101717638 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚761366698913792 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 868220928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688939753647966 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚985 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400783695 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚697192340027392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10044341640192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339635712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚678931715868672 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 868220928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 656375808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10007366301970 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159013168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 trainer/global_step‚11999 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 174340616 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226423015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57295532 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 % _timestamp‚1.7884260431304832e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚469 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚697192358030336 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159013637 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚467 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10536593839104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚697192358030336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 _runtime‚9542.868403342 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚707237356046336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400783695 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57295999 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚430 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 9326034944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400763629 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226423015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 153124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 8838745088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚678931732908032 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÙÞ^ó»2û*- _wandbruntime‚9542  _runtime‚9542.I[6^ü*Ø /loss/FBt1/detection_loss‚2.1809960454702377 _runtime‚9542.869882052 0loss/FRt1/assignment_loss‚3.0353101022700044 2loss/SRqqt1/assignment_loss‚1.3859236818240133 1loss/SRqqt3/detection_loss‚1.6586058521479892 0loss/FBt1/assignment_loss‚0.2936400492520382 1loss/FBt2/assignment_loss‚0.11417903994055223 /loss/FBt4/detection_loss‚0.6512209979413698 %loss/total_loss‚40.36323880416193 trainer/global_step‚11999 1loss/FBt3/assignment_loss‚0.12749518662894843 /loss/FRt2/detection_loss‚2.7590015146040145 /loss/FRt3/detection_loss‚2.0467761523193784 .loss/FRt4/detection_loss‚1.681940395385027 0loss/FRt4/assignment_loss‚1.1035002532758211 2loss/SRqqt3/assignment_loss‚1.3617717528127526 1loss/FBt4/assignment_loss‚0.16904086862467332 1loss/SRqqt4/detection_loss‚0.9977149903064683 .loss/FRt1/detection_loss‚2.220323069298521 $ _timestamp‚1.788426043132422e+09 _step‚986 /loss/FRt2/assignment_loss‚3.134033137752164 2loss/SRqqt2/assignment_loss‚2.1474859875937304 /loss/FBt3/detection_loss‚2.4232297092676163 epoch‚9 0loss/FRt3/assignment_loss‚2.8284963877112776 2loss/SRqqt4/assignment_loss‚0.8791250189256278 1loss/SRqqt1/detection_loss‚1.9391289706840071 .loss/FBt2/detection_loss‚2.889499055014716 1loss/SRqqt2/detection_loss‚2.3348005851109823%Æícý*Ý /loss/FBt3/detection_loss‚2.4232297092676163 epoch‚9 0loss/FRt3/assignment_loss‚2.8284963877112776 2loss/SRqqt4/assignment_loss‚0.8791250189256278 1loss/SRqqt1/detection_loss‚1.9391289706840071 .loss/FBt2/detection_loss‚2.889499055014716 1loss/SRqqt2/detection_loss‚2.3348005851109823 /loss/FBt1/detection_loss‚2.1809960454702377 _runtime‚9542.869882052 0loss/FRt1/assignment_loss‚3.0353101022700044 2loss/SRqqt1/assignment_loss‚1.3859236818240133 1loss/SRqqt3/detection_loss‚1.6586058521479892 0loss/FBt1/assignment_loss‚0.2936400492520382 1loss/FBt2/assignment_loss‚0.11417903994055223 /loss/FBt4/detection_loss‚0.6512209979413698 %loss/total_loss‚40.36323880416193 trainer/global_step‚11999 1loss/FBt3/assignment_loss‚0.12749518662894843 /loss/FRt2/detection_loss‚2.7590015146040145 /loss/FRt3/detection_loss‚2.0467761523193784 .loss/FRt4/detection_loss‚1.681940395385027 0loss/FRt4/assignment_loss‚1.1035002532758211 2loss/SRqqt3/assignment_loss‚1.3617717528127526 1loss/FBt4/assignment_loss‚0.16904086862467332 1loss/SRqqt4/detection_loss‚0.9977149903064683 .loss/FRt1/detection_loss‚2.220323069298521 $ _timestamp‚1.788426043132422e+09 _step‚986 /loss/FRt2/assignment_loss‚3.134033137752164 2loss/SRqqt2/assignment_loss‚2.1474859875937304Ú SIÝ"þ*‚(:‡ ÇîäÔ„É€~ gpu.0.u®†–pgpu‚55.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.083' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.2166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lm¶Hoÿ*‚(:ä ÇîäÔ–甀 disk.drbd1716.out‚0& disk.nvme0n1.out‚40552.37060546875# disk.dm-0.out‚40552.37060546875 disk.rbd1.in‚0 cpu‚25.33427993349091 disk.drbd2080.out‚0 disk.dm-1.out‚ 490.0546875 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 490.0546875$ memory_percent‚93.72274875640869* disk./.usagePercent‚17.576710558373325 disk.rbd0.out‚0) proc.memory.availableMB‚ 1028.46484375 disk.dm-1.in‚ 126.77734375 disk.drbd2266.in‚0 disk.sda.out‚ 490.0546875) proc.memory.percent‚16.43803119659424 proc.cpu.threads‚17 disk.dm-0.in‚4.6875 disk.sda.in‚ 126.77734375 network.sent‚ 2.9123575e+07 network.recv‚ 3.44054e+06% disk./.usageGB‚261.90568923950195 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚4.90625 disk.sda1.out‚0 disk.nvme0n1.in‚4.6875( disk.nvme0c0n1.out‚40552.37060546875 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2693.20703125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0!rö’€+‚(:‡ ÖîäÔ®àŸ~ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.049' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4098 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ucpk+‚(:à ÖîäÔñ¡€ disk.drbd2265.out‚0 cpu‚25.333861240888865 disk.sda1.in‚0 disk.rbd0.out‚0 disk.drbd1716.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.57732856056098 disk.dm-0.in‚4.6875$ memory_percent‚93.72291564941406) proc.memory.percent‚16.43803119659424 network.sent‚ 2.9232424e+07 disk.rbd1.in‚0 disk.rbd1.out‚4.90625 disk.drbd1716.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚40628.7607421875" disk.dm-0.out‚40628.7607421875# proc.memory.rssMB‚ 2693.20703125 disk.nvme0n1.in‚4.6875 disk.drbd2266.in‚0 network.recv‚ 3.446707e+06 disk.sda.out‚ 490.70703125 disk.drbd2080.in‚0 disk.dm-1.in‚ 126.77734375$ disk./.usageGB‚261.9148979187012 disk.sda.in‚ 126.77734375 disk.sda1.out‚0 disk.nvme0c0n1.in‚4.6875 disk.sda3.in‚ 126.77734375 disk.sda3.out‚ 490.70703125 disk.dm-1.out‚ 490.70703125% proc.memory.availableMB‚ 1028.4375% disk.nvme0n1.out‚40628.7607421875 disk.sda2.in‚0.z³ 2‚+- _wandbruntime‚9579  _runtime‚9579‰\q×ƃ+À _runtime‚9579.138224133 _step‚987 % lr-AdamW/pg1‚5.90942707586136e-09 % lr-AdamW/pg2‚5.90942707586136e-09 trainer/global_step‚12049 % _timestamp‚1.7884260794007194e+09§%®Ë„+Å % lr-AdamW/pg1‚5.90942707586136e-09 % lr-AdamW/pg2‚5.90942707586136e-09 trainer/global_step‚12049 % _timestamp‚1.7884260794007194e+09 _runtime‚9579.138224133 _step‚987ÛÕ»Ù:2…+- _wandbruntime‚9579  _runtime‚9579h0˜$ñ'†+ëO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10085164380672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 656362496 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 175027398 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 159638431 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚908 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚30 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 159429632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 183369728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 $ _timestamp‚1.788426079401369e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57520987 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472526 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 _step‚988 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578903079424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚774966647568384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159638857 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57520563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508018176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710038146949120 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 9955586048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 9643699025 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 9326034944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 654437756 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚844 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚699953656933888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339635712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 10194255872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚414 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10084508018176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57520563 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚764387585059328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚844 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 10194255872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10084508018176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚414 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2605 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚844 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 863815680 $ _timestamp‚1.788426079401369e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 9326034944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2605 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚699953656933888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 656362496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402378051 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 10194255872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 868220928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚426 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚430 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 402358011 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 9148669952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10085164380672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57520987 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚691669403472526 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10048039644170 _step‚988 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 863815680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10578903079424 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10047385206414 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚889 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚774966647568384 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚414 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 674365440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175027400ÜWO•D2ˆ+- _wandbruntime‚9579  _runtime‚9579MšûÂ!‰+õM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57525470 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579787812864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227349067 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159651069 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57525033 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227369131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _runtime‚9579.834957097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175041416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚12049 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175041414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159651508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016 >6DeviceStatsMonitor.ƒ"GÏ9on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788426080096969e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579936743424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993794560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700010009972736!Ók¸'Š+úM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 9148669952 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚844 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10085327446016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚844 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 671477324 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚430 $ _timestamp‚1.788426080096969e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚437 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10579936743424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚430 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710095993794560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700010009972736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10085983821824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 10194255872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57525470 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175041414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710095319415808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 9955586048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10579787812864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 654437964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 9326034944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227349067 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159651069 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 868220928 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚414 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700010009972736 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 102126036 _step‚989 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102126038 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 10194255872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175041416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710095993794560 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 863815680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57525033 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10085983821824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227369131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764449235796992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 656375808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚30 _runtime‚9579.834957097 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402410547 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175041416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2605 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10048856079342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚889 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 9643699025 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 863815680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚764449211856896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚844 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚414 trainer/global_step‚12049 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 175041414 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159651508 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚691725124038050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227369131 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10085327446016 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 674378752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227349067 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 268325376 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚414 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚685602481833984 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚704068784934912 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57936072 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10099354775594 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚151 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581978 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 102771452 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 132992512 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10098863861202 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 228466407 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10136675672064 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚704068802937856 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚153 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚695701853648938 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 507953752 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 10194255872 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 863815680 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 228466407 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 8838745088 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 10194255872 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 10194255872 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 868220928 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 510833152 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10632512625664 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 176131548 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 863815680 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10136182841856 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 176131548 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 102771454 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 9955586048 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚714205478609920 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚844 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 228450430 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚889 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚768771152305152 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚685602498873344 # _timestamp‚1.78842615771847e+09 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 404581978 _runtime‚9657.456397331 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 268325376 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚695701345695186 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 868220928 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 510833152 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 LDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚908 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚714205478609920 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 9326034944 trainer/global_step‚12089 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚10136182841856 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2605 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚685602481833984 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚844 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚704068784934912 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚57936072 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10099354775594 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 176131550 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 492830208 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚151 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚844 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 404581978 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2605 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 9148669952 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 102771452 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 132992512 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 492830208 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 163111424 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 9326034944 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10098863861202 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 9955586048 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 228466407 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 9148669952 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10136675672064 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2ßhŸi2›+- _wandbruntime‚9658  _runtime‚9658kvýc)œ+ÝR XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 132992512 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚779441905790976 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚8447&òh)+âR JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2605 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2605 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10632877752320 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 10194255872 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 492830208 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 163111424 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚685637792632832 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚704105632966656 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10632986804736 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 490914392 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 10194255872 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚430 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚414 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 507953752 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚844 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚714242162932224 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404600086 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚430 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404616065 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 132992512 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚779441905790976 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚844 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 228473709 JBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚908 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 8838745088 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.573585569858551 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30090996623039246 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36666664481163025 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03537362068891525 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.504982054233551 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5172719955444336 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5708566308021545 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08227112889289856 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5702034831047058 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020674937404692173 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47777777910232544 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.557060182094574 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4133603274822235 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979490041732788 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34315475821495056 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_5_of_6‚0.01923076994717121 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900011420249939 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 particle/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5603218078613281 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.539097785949707 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06413868814706802 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03420886769890785 *jet/accuracy_1_of_3‚0.8933826684951782 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 _runtime‚9671.199922596 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *particle/sensitivity‚0.309223473072052 jet/accuracy_7_of_7‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16372475028038025 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚12089 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09015334397554398 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2331349104642868 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1958235651254654 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12521149218082428 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12490642070770264 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9604902863502502 particle/accuracy_4_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899956583976746 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666269302368 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.608134925365448 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.625491738319397 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.019965793937444687 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_4_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04105030745267868 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0172488484531641 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5058478713035583 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3734370470046997 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4623985290527344 jet/accuracy_6_of_6‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6216636300086975 &particle/f_score‚0.389476478099823 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13676884770393372 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4292635917663574 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12010268867015839 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5890804529190063 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1650538295507431 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5488815903663635 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5545693635940552 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5352670550346375 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11229048669338226 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4924837350845337 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.094150610268116 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9909855723381042 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 jet/accuracy_1_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 % _timestamp‚1.7884261714602344e+09 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46075180172920227 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5326948165893555 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376544773578644 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07459331303834915 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 5"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.550048828125 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 particle/accuracy_2_of_7‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5140044689178467 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013941236771643162 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13248729705810547 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5649020075798035 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23269079625606537 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26415953040122986 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10914579778909683 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3560040295124054 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8487309217453003 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4199828505516052 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3006203770637512 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6291666030883789 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5037356615066528 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5781498551368713 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,jet/accuracy_4_of_4‚0.005652218125760555 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5250245928764343 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9844664335250854 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3365571200847626 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5871288180351257 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7851694822311401 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26046425104141235 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5885140299797058 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17908498644828796 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2661532759666443 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09062085300683975 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5745500326156616 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 (loss/val_total_loss‚42.4537284705472 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2613208591938019 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.505938708782196 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_5‚0.5118215084075928 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012960828840732574 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2952413260936737 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41747406125068665 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20774246752262115 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5779516100883484 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08588436245918274 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5731862783432007 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.49357306957244873 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5001025795936584 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.583125114440918 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 6validation_average_jet_accuracy‚0.4565330147743225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31647545099258423 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847596883773804 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6404296159744263 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888890504837036 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20319239795207977 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3899471163749695 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666269302368 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 _step‚993 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5729262232780457 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.5030009150505066 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08074421435594559 epoch‚9 *jet/accuracy_2_of_4‚0.3846408724784851 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.604057788848877 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5156540870666504 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.699272096157074 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5967323184013367 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 jet/accuracy_2_of_7‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12448041886091232 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008046514354646206 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20291486382484436 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5855318307876587 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8124809265136719 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888114929199219 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1705438494682312 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011111111380159855 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917669892311096 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24446053802967072 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 +jet/accuracy_2_of_3‚0.24869190156459808 *jet/accuracy_1_of_4‚0.9313787817955017 +jet/accuracy_3_of_4‚0.07074126601219177 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896060228347778 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24719160795211792 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12881878018379211 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.021604938432574272 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5659377574920654 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5122694373130798 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09005847573280334 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3635313808917999 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4978114068508148 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798515439033508 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030352234840393066 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3564814627170563 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 particle/accuracy_4_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4650246798992157 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515049397945404 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 * Purity/2FRt3SRqqt0FBt/FRt_purity‚0.125 +jet/accuracy_3_of_5‚0.15350742638111115 particle/accuracy_3_of_6‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5731862783432007 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0891576036810875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5651893615722656 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4506579041481018 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5806547403335571 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6256409883499146 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19243410229682922 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 particle/accuracy_3_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38154855370521545 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41160711646080017 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5856481790542603 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16175760328769684 >&Purity/2FRt*SRqqt1FBt/evente)&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982101321220398 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925991892814636 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09516417980194092 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17353199422359467 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5580289959907532 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.7105262875556946 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913349747657776 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1375117152929306 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5480740070343018 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.174052432179451 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2718259394168854 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883156418800354 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16372475028038025 particle/accuracy_2_of_4‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11580459773540497 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9485734105110168 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1600671410560608 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5695531964302063 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17258641123771667 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18697358667850494 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.052681997418403625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28148147463798523 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.024014778435230255 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019679220393300056 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5866276621818542 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0á þåh¢+¦ @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2613208591938019 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 8"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.505938708782196 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 *jet/accuracy_2_of_5‚0.5118215084075928 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4479166865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.012960828840732574 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0416666679084301 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2952413260936737 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.41747406125068665 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 +jet/accuracy_4_of_6‚0.03846153989434242 & Purity/*FRt*SRqqt4FBt/FRt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20774246752262115 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5779516100883484 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08588436245918274 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833335518836975 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5731862783432007 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.49357306957244873 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5001025795936584 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.583125114440918 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 6validation_average_jet_accuracy‚0.4565330147743225 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31647545099258423 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847596883773804 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6404296159744263 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_2_of_2‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888890504837036 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20319239795207977 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3899471163749695 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6041666269302368 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 _step‚993 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5729262232780457 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.5030009150505066 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08074421435594559 epoch‚9 *jet/accuracy_2_of_4‚0.3846408724784851 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.604057788848877 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.5156540870666504 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.699272096157074 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5967323184013367 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7272727489471436 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5333333611488342 jet/accuracy_2_of_7‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12448041886091232 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008046514354646206 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20291486382484436 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5855318307876587 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8124809265136719 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888114929199219 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1705438494682312 & Purity/1FRt*SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011111111380159855 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917669892311096 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.24446053802967072 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9828309416770935 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3888888955116272 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 +jet/accuracy_2_of_3‚0.24869190156459808 *jet/accuracy_1_of_4‚0.9313787817955017 +jet/accuracy_3_of_4‚0.07074126601219177 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740737318992615 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896060228347778 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0008818342466838658 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24719160795211792 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12881878018379211 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.021604938432574272 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5659377574920654 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.5122694373130798 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09005847573280334 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3635313808917999 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4978114068508148 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4197530746459961 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9798515439033508 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.030352234840393066 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3564814627170563 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 particle/accuracy_4_of_6‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4650246798992157 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515049397945404 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9937641620635986 * Purity/2FRt3SRqqt0FBt/FRt_purity‚0.125 +jet/accuracy_3_of_5‚0.15350742638111115 particle/accuracy_3_of_6‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5731862783432007 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0891576036810875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5651893615722656 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4506579041481018 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5806547403335571 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6256409883499146 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.19243410229682922 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9785714745521545 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 particle/accuracy_3_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38154855370521545 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.41160711646080017 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5856481790542603 particle/accuracy_7_of_7‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16175760328769684 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5320513248443604 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5702101588249207 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1331225037574768 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.09508834779262543 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982101321220398 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9925991892814636 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09516417980194092 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17353199422359467 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5580289959907532 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.7105262875556946 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913349747657776 jet/accuracy_5_of_5‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_5_of_6‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1375117152929306 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5480740070343018 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.174052432179451 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2718259394168854 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9883156418800354 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16372475028038025 particle/accuracy_2_of_4‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11580459773540497 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9485734105110168 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9798850417137146 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1600671410560608 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5695531964302063 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17258641123771667 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18697358667850494 9 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.052681997418403625 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28148147463798523 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.024014778435230255 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019679220393300056 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5866276621818542 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_4_of_5‚0.010906298644840717 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09937373548746109 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12277895957231522 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6018311977386475 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5669973492622375 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.004098360426723957 particle/accuracy_3_of_3‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5560547709465027 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.15090090036392212 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6047681570053101 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5529157519340515 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.573585569858551 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30090996623039246 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36666664481163025 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03537362068891525 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.504982054233551 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5172719955444336 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5708566308021545 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08227112889289856 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3958333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07619047164916992 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5702034831047058 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020674937404692173 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.47777777910232544 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.557060182094574 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4133603274822235 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979490041732788 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34315475821495056 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.40909090638160706 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4479166865348816 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_5_of_6‚0.01923076994717121 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 jet/accuracy_5_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9900011420249939 particle/accuracy_3_of_4‚1 particle/accuracy_1_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 particle/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5603218078613281 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.539097785949707 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06413868814706802 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.6875 :"Purity/1FRt2SRqqt0FBt/event_purity‚0.00266400258988142 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03420886769890785 *jet/accuracy_1_of_3‚0.8933826684951782 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 _runtime‚9671.199922596 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *particle/sensitivity‚0.309223473072052 jet/accuracy_7_of_7‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16372475028038025 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 trainer/global_step‚12089 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.09015334397554398 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2331349104642868 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1958235651254654 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12521149218082428 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.12490642070770264 @&Purity/3FRt2SRqqt0FBt/event@gPd*_proportion‚0.0024801588151603937 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9604902863502502 particle/accuracy_4_of_5‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9899956583976746 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6041666269302368 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.01666666753590107 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.608134925365448 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.625491738319397 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.019965793937444687 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 jet/accuracy_4_of_7‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0031746032182127237 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6764705777168274 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04105030745267868 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0172488484531641 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.5058478713035583 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3734370470046997 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4623985290527344 jet/accuracy_6_of_6‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.75 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.6216636300086975 &particle/f_score‚0.389476478099823 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13676884770393372 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4292635917663574 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12010268867015839 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5890804529190063 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1650538295507431 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5488815903663635 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5545693635940552 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5352670550346375 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11229048669338226 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4924837350845337 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0416666679084301 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11702127754688263 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.094150610268116 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9909855723381042 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.7222222089767456 jet/accuracy_1_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 % _timestamp‚1.7884261714602344e+09 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 particle/accuracy_5_of_7‚1 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46075180172920227 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5326948165893555 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22376544773578644 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2649425268173218 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07459331303834915 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 5"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.550048828125 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 particle/accuracy_2_of_7‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9884405732154846 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5140044689178467 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013941236771643162 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13248729705810547 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5649020075798035 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23269079625606537 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26415953040122986 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5617284178733826 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.10914579778909683 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3560040295124054 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8487309217453003 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4199828505516052 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3006203770637512 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6291666030883789 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5037356615066528 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9795321226119995 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5781498551368713 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,jet/accuracy_4_of_4‚0.005652218125760555 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5250245928764343 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9844664335250854 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3365571200847626 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5871288180351257 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7851694822311401 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26046425104141235 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5885140299797058 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17908498644828796 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2661532759666443 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09062085300683975 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5745500326156616 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 (loss/val_total_loss‚42.4537284705472 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_2_of_3‚1 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_7‚1áÈ¿Tz£+‚(:„ ¿ïäÔ’Àƒ~ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚12.504959106445312* gpu.0.memoryAllocatedBytes‚ 10741678080 gpu.0.temp‚32.0 gpu.0.powerWatts‚99.645' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.929 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wRDck¤+‚(:à ¿ïäÔ›Ô¦€ disk.sda3.in‚ 126.98046875 disk.rbd1.out‚4.90625 cpu‚25.362356935752 disk.dm-0.in‚4.6875" disk.dm-0.out‚41071.8779296875 disk.drbd2266.out‚0' proc.memory.availableMB‚ 477.3515625 network.sent‚ 2.9478387e+07 disk.nvme0c0n1.in‚4.6875 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 494.48828125 disk.sda1.out‚0$ disk./.usageGB‚262.4421157836914% disk.nvme0n1.out‚41071.6279296875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚41071.6279296875$ memory_percent‚97.08647727966309 proc.cpu.threads‚17 disk.nvme0n1.in‚4.6875 disk.sda3.out‚ 494.48828125 disk.dm-1.in‚ 126.98046875 disk.dm-1.out‚ 494.48828125 disk.sda.in‚ 126.98046875" proc.memory.rssMB‚ 3245.3359375 network.recv‚ 3.479522e+06 disk.sda2.in‚0* proc.memory.percent‚19.807958602905273 disk.rbd0.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd1716.out‚0* disk./.usagePercent‚17.612710593845726mAÑ>R¥+ 0h3k07stj@ ¿ïäÔˆ„¿0`Trainer.fit` stopped: `max_epochs=10` reached. 8hšþŸ¦+ 0h3k07stjŒ ÀïäÔð…€ÄzEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:54 • 0:00:00 1.36it/s v_num: 07st½# ]%§+ 0h3k07stj ÀïäÔì’Ä •„åk2¨+- _wandbruntime‚9677  _runtime‚9677 à >©+‚(’ÍK